MEASUREMENT  OF  COMPUTER 


USER  SATISFACTION 


A Dissertation  Presented  in  Partial  Fulfillment 
of  the  Requirements  for  the  Degree 
v Doctor  of  Philosophy 


ARIZONA  STATE  UNIVERSITY 


August,  1977 


SECURITY  CLASS»FICATIQN  OP  this  PAGE  (When  rrhtm  kCnty.yJ) 

I * REPORT  DOCUMENTATION  PAGE" 


(Ann  ^ ci^s-i  f ‘ j_sB. 

— ~~^C-  *•  title  *■•>■''■»-'  1 1 

f£\  ) Measurement  of  Computer  User  Satisfaction  / 


READ  INSTRUCTIONS 
BEFORE  COMPLETING  FORM 


« PM lOp  COVERED 


Dissertation-  

« PERFORMING  ORG  REPORT  NUMBER 


1 7.  AUTHORfT) 


8.  CONI  PACT  OR  GRANT  NUMBERf*) 


Captain) Sammy  Wl^earson 


19.  performing 


10.  PPOGPAW  ELEMENT.  PROJECT.  TASK 
APE  A A WORK  UNIT  NUMBERS 


AFIT  Student  at  Arizona  State  University, 

Tempe  AZ 

11.  CONTROLLING  OFFICE  NAME  AND  ADDRESS  -■ fU.  BEBftBT  ftATft 

AFIT/CI  (/  //  Augmt  *9  77  j 

WPAFB  OH  45433  X-.i*.  iimii’iiPPrtft 

314  Pages 

U.  MONITORING  AGENCY  NAME  8 ADDRESSfT/  dllterent  Irom  Controlling  Olllce)  IS.  SECURITY  CLASS,  (ol  Ihle  report) 


Y-9J3& 


Unclassified 

15a.  OECLASSIRCATION/ DOWNGRADING 
SCHEDULE 


[ 16.  DISTRIBUTION  STATEMENT  (of  thle  Report) 


Public  Releaser  Distribution  Unlimited 


^DISjAteUTig^^^^^M^ENT  (ol  I ha  ebelract  entered  In  Block  20,  II  dlllerent  Irom  Report) 


^u^lementa^rJYesI.  APPROVED  FOR  PUBLIC  RELEASE  AFR  19017. 

JERRAb) F . GUESS , Captain , USAF 
Director  of  Information,  AFIT 


1 19.  KEYWORDS  (Continue  on  reveree  aide  if  necessary  end  Identify  by  block  number) 


I 20.  ABSTRACT  (Continue  on  reverse  side  If  necessary  and  Identify  by  block  number ) 


DD  .ET*  1473  EDITION  OF  1 NOV  65  IS  OBSOL£*t 


COITION  OF  1 NOV  65  IS  OBSOl 


SECURITY  CLASSIFICATION  OF  THIS  ° AGt  (When  Date  Entered) 


...  INSTRUCTIONS  FOR  PREPARATIpN  OF  REPORT  DOCUMENTATION  PACE 

RESPONSIBILITY . The  controlling  DoD  office  will  be  responsible  for  completion  of  the  Report  Documentation  Page,  DD  Form  1473,  in 
all  technical  reports  prepared  by  or  for  DoD  organizations. 

CLASSIFICATION.  Since  this  Report  Documentation  Page,  DD  Form  1473,  is  used  in  preparing  announcements,  bibliographies,  and  data 
banks,  it  should  be  unclassified  if  possible.  If  a classification  is  required,  identify  the  classified  items  on  the  page  by  the  appropriate 
symbol. 

COMPLETION  GUIDE 

General.  Make  Blocks  1,  4,  5,  6,  7,  11,  13,  15,  and  16  agree  with  the  corresponding  information  on  the  report  cover.  Leave 
Blocks  2 and  3 blank. 

Block  1.  Report  Number.  Enter  the  unique  alphanumeric  report  number  shown  on  the  cover. 

Block  2.  Government  Accession  No.  Leave  Blank.  This  space  is  for  use  by  the  Defense  Documentation  Center. 

Block  3.  Recipient's  Catalog  Number.  Leave  blank.  This  space  is  for  the  use  of  the  report  recipient  to  assist  in  future 
retrieval  of  the  document. 

Block  4.  Title  and  Subtitle.  Enter  the  title  in  all  capital  letters  exactly  as  it  appears  on  the  publication.  Titles  should  be 
unclassified  whenever  possible.  Write  out  the  English  equivalent  for  Greek  letters  and  mathematical  symbols  in  the  title  ("see 
•*  Abstracting  Scientific  and  Technical  Reports  of  Detense-sponsored  RDT/E,  "AD-667  000).  If  the  report  has  a subtitle,  this  subtitle 
should  follow  the  main  title,  be  separated  by  a comma  or  semicolon  if  appropriate,  and  be  initially  capitalized.  If  a publication  has  a 
title  in  a foreign  language,  translate  the  title  into  English  and  follow  the  English  translation  with  the  title  in  the  original  language. 

Make  every  effort  to  simplify  the  title  before  publication. 

Block  5.  Type  of  Report  and  Period  Covered.  Indicate  here  whether  report  is  interim,  final,  etc.,  and,  if  applicable,  inclusive 
dates  of  period  covered,  such  as  the  life  of  a contract  covered  in  a final  contractor  report. 

Block  6.  Performing  Organization  Report  Number.  Only  numbers  other  than  the  official  report  number  shown  in  Block  1,  such 
as  series  numbers  for  in-house  reports  or  a contractor/grantee  number  assigned  by  him,  will  be  placed  in  this  space.  If  no  such  numbers 
are  used,  leave  this  space  blank. 

Block  7.  AuthorfsL  Include  corresponding  information  from  the  report  cover.  Give  the  namefs)  of  the  author/*,)  in  conventional 
order  ( for  example,  John  R.  Doe  or,  if  author  prefers,  J.  Robert  Doe).  In  addition,  list  the  affiliation  of  an  author  if  it  differs  from  that 
of  the  performing  organization. 

Block  8.  Contract  or  Grant  Numberfs,).  For  a contractor  or  grantee  report,  enter  the  complete  contract  or  grant  numberfs,)  under 
which  the  work  reported  was  accomplished.  Leave  blank  in  in-house  reports. 

Block  9.  Performing  Organization  Name  and  Address.  For  in-house  reports  enter  the  name  and  address,  including  office  symbol, 
of  the  performing  activity.  For  contractor  or  grantee  reports  enter  the  name  and  address  of  the  contractor  or  grantee  who  prepared  the 
report  and  identify  the  appropriate  corporate  division,  school,  laboratory,  etc.,  of  the  author.  List  city,  state,  and  ZIP  Code. 

Block  10  Program  Element,  Project,  Task  Area,  and  Work  Unit  Numbers.  Enter  here  the  number  code  from  Die  applicable 
Department  of  Defense  form,  such  as  the  DD  Form  1498,  “Research  and  Technology  Work  Unit  Summary”  or  the  DD  Form  1634. 

“Research  and  Development  Planning  Summary,”  which  identifies  the  program  element,  project,  task  area,  and  work  unit  or  equivalent 
under  which  the  work  was  authorized. 

Block  11.  Controlling  Office  Name  and  Address.  Enter  the  full,  official  name  and  address,  including  office  s_ymbol,  of  the 
controlling  office.  (Equates  to  funding/ sponsoring  agency.  For  definition  see  DoD  Directire  5200.20,  "Distribution  Statements  on 
Technical  Documents.”) 

Block  12.  Report  Date.  Enter  here  the  day,  month,  and  year  or  month  and  year  as  shown  on  the  cover. 

Block  13.  Number  of  Pages.  Enter  the  total  number  of  pages. 

Block  14.  Monitoring  Agency  Name  and  Address  (if  different  from  Controlling  Office).  For  use  when  the  controlling  or  funding 
office  does  not  directly  administer  a project,  contract,  or  grant,  but  delegates  the  administrative  responsibility  to  another  organization. 

Rlocks  15  & 15a  Security  Classification  of  the  Report:  Declasaification/Downgrading  Schedule  of  the  Report.  Enter  in  15 
the  highest  classification’ of  the  report.  If  appropriate,  enter  in  15a  the  dectassification/downgrading  schedule  of  the  report,  using  the 
abbreviations  *or  declassification/ downgrading  schedule,  listed  in  paragraph  4-207  of  DoD  5200. 1-R. 

Block  16.  Distribution  Statement  of  the  Report.  Insert  here  the  applicable  distribution  statement  of  the  report  from  DoD 
Directive  5200.20,  “Distribution  Statements  on  Technical  Documents." 

„ , „ c,., ,ni  lh-  abstract  entered  in  Block  20,  if  different  from  the  distribution  statement  of  the  report). 

Insert  hel^pUc.^  D°D  52°°-20'  1,8  T~hniC-'  ^ 

uments. M 

Block  18  Supplementary  Notes.  Enter  information  not  included  elsewhere  but  useful,  such  as:  Prepared  in  cooperation  with 
. Translation-^  (or  by)  . . . Presented  at  conference  of  ...  To  be  published  in  . . . 

1.10*  K.u  Wnrds  Select  terms  or  short  phrases  that  identify  the  principal  subjects  covered  in  the  report,  and  are 
sufficienW^fic  .Vnd  precise  ^be  u.TdV.  index  entries  for  cataloging,  conforming  to  standard  terminology.  The  DoD  "Thesaurus 
of  Engineering'and  Scientific  Terms”  (TEST),  AD-672  000.  can  be  helpful. 

r..  U on  Ahttr.ri  The  abstract  should  be  a brief  (not  to  exceed  200  words)  factual  summary  of  the  most  significant  Informa- 
tion contfl^TTthe  report.’  If  possible,  the  abstract  of  - 

[rhere.^Fo^inforrnation  o'n>pre*paringeabstracts*aee*"Abstrac^ngr{^Mitifrc*and  T.Tnlc‘.,  Report.  5 SeJen.e-Spon.ored  RDT*E,” 
AD-667  000. 


* U.S.  SOVECXMENT  FRINTINS  OFFICE  : 1973-729-091/1431  3-1 


% 


MEASUREMENT  OF  COMPUTER 
USER  SATISFACTION 

by 

Sammy  Wray  Pearson 

has  been  approved 
July,  1977 


APPROVED: 


bepartment  Chairperson 


Col, 

Dean,  Graduate  College 


ACKNOWLEDGMENTS 


I wish  to  express  my  appreciation  to  the  numerous 
people  who  have  contributed  to  the  successful  completion  of 
this  work.  I am  especially  grateful  to: 

Professor  James  E.  Bailey,  for  the  direction  he 
provided  to  this  research  and  for  the  generous  sharing  of 
his  time  and  counsel. 

Professors  D.D.  Bedworth,  C.D.  Hoyt,  W.E.  Lewis,  and 
W.C.  Moor,  for  their  active  participation  and  free  exchange 
of  opinions  and  ideas. 

Bill  Romaine  and  Tom  Webster,  fellow  students,  for 
sharing  the  joys  and  the  tribulations  of  the  educational 
experience. 

Mr.  and  Mrs.  Howard  S.  Pearson,  my  parents,  for  their 
lifelong  encouragement  and  inspiration  in  all  of  my 
endeavors . 

Finally,  my  wife  Barbara,  for  creating  the  environment 
necessary  for  success  with  her  love,  patience,  and  constant 
good  cheer. 


iv 


V 


Chapter 

2. 


Page 


Advantages  of  the  Semantic 

Differential  55 

SUMMARY 59 

3.  DOMAIN  OF  THE  USER  SATISFACTION 

CONSTRUCT 61 

LITERATURE  REVIEW 62 

Operational  Definitions  of  the 

Factors 73 

Independent  Review  98 

EMPIRICAL  TEST  OF  THE  FACTOR  LIS'”.  . 100 

Sample  Selection  Process  105 

Interview  Procedure 110 

Data  Extractions 113 

Test  Results 119 

SUMMARY 124 

4.  CONSTRUCTION  OF  THE  MEASUREMENT 

INSTRUMENT 126 

SELECTION  OF  SCALES 127 

Selection  Procedure 129 

Weighting  Function  Scale  132 

Internal  Consistency  Scale  ....  133 

FORMAT  OF  THE  INSTRUMENT 133 

Number  of  Scale  Intervals 136 

Choice  of  Adverb  Modifiers  ....  138 

Order  of  the  Scales 139 

vi 


- 


' * JjCM 


Chapter  Page 

4. 

Order  of  the  Factors 140 

Instruction  Set 141 

Assembly 141 

PRELIMINARY  EVALUATION 142 

SUMMARY 143 

5.  EVALUATION  OF  THE  MEASUREMENT 

INSTRUMENT 144 

ADMINISTRATION  AND  SCORING 145 

RELIABILITY 151 

VALIDITY 154 

Content  Validity 156 

Predictive  Validity  162 

Construct  Validity 172 

SUMMARY 181 

6.  SUMMARY,  CONCLUSIONS,  AND 

RECOMMENDATIONS  183 

SUMMARY 183 

CONCLUSIONS 186 

RECOMMENDATIONS  190 

BIBLIOGRAPHY 193 

APPENDICES 

3.1  FACTOR  IDENTIFICATION  BY  STUDY 212 

3.2  CORRESPONDENCE  WITH  SUBJECTS 215 

3.3  BACKGROUND  QUESTIONNAIRE  AND 

CONSENT  STATEMENT  221 


vii 


r 


APPENDICES  Page 

13.4  INTERVIEW  GUIDELINES  224 

3.5  DATA  EXTRACTION  SYLLABUS 228 

3.6  FACTORS  MENTIONED  BY  SUBJECT  234 

3.7  FACTOR  RANKINGS  BY  SUBJECT  238 

r 

3.8  FACTOR  LISTING  AND  DEFINITIONS  242 

4.1  USER  SATISFACTION  MEASUREMENT 

INSTRUMENT 248 

4.2  INSTRUCTIONS  FOR  ADJECTIVE  PAIR 

SELECTION 273 

4.3  ADJECTIVE  PAIR  SELECTIONS 296 

5.1  FORTRAN  PROGRAM  FOR  DATA 

SUMMARIZATION 301 

5.2  FORTRAN  PROGRAM  FOR  ANALYSIS  OF 

VARIANCE 303 

5.3  SAMPLE  FACTOR  ANALYSIS  OUTPUT 306 

5.4  SELECTION  OF  CLASSIFIED  USERS 310 

5.5  FACTOR-DIMENSION  MATRIX 313 


5.5  FACTOR-DIMENSION  MATRIX 313 


I 


T 


LIST  OF  TABLES 


Table  Page 

2.1  Common  Measurement  Scales  20 

2.2  Example  of  Thurstone  Type  Scale  Items  ...  26 

2.3  Example  of  Likert  Type  Scale  Items 29 

2.4  Example  of  Guttman  Scalogram 31 

2.5  Example  of  Semantic  Differential 

Scales 34 

2.6  Sample  EPA  Adjective  Pairs 35 

3.1  Studies  Reviewed  for  Factor  Identification.  69 

3.2  Factors  Identified  from  Studies  74 

3.3  Factors  Affecting  User  Satisfaction  ....  101 

3.4  Sample  Distribution  109 

3.5  Data  Extraction  Comparison 117 

3.6  Distribution  of  Factor  Rankings  122 

5.1  Satisfaction  Scores  150 

5.2  Reliability  Coefficients  by  Factor 155 

5.3  Correlation  Coefficients  Between  Factor 

Scales 159 

5.4  Scale  Factor  Loadings  161 

5.5  Scale  Range  Between  Means  163 

5.6  Subject  Satisfaction  Scores  165 

5.7  Response  Group  Mean  Scores 168 

5.8  Satisfaction  Scale  and  Factor  Score 

Correlation 169 


ix 


LIST  OF  ILLUSTRATIONS 

Model  of  Factor  Categories 

Syllabus  Procedures  for  Interview 
Evaluations  

Satisfaction  Scale 

Scoring  Procedures 

Sample  ANOVA  Table 

Subject  Self-Addressed  and  Measured 
Satisfaction  Scores  


xi 


P*TT'  ' 9 '•  ,J  - — i-j— T'.-v- 


Chapter  1 


INTRODUCTION 


In  the  national  economy,  the  budget  for  Electronic  Data 
Processing  (EDP)  services  for  1974  was  approximately  $28 
billion  (McLaughlin,  1974) . The  growth  rate  for  this  activ- 
ity has  been  averaging  8 to  10  percent  per  year  in  the  early 
1970' s.  The  effect  of  this  large  expenditure  on  the  affect- 
ed organizations  has  been  mixed.  For  example,  Staats  (1973) 
credited  automation  and  management  improvements  with  wage 
reductions  of  over  $200  million  annually  for  the  period 
1967-72  in  government  organizations  that  maintain  individual 
citizens'  records.  A private  firm  saved  $750,000  a year  in 
"hard"  dollars  by  changing  from  batch  to  on-line  data  pro- 
cessing (Rickert,  1975) . 

Yet,  these  apparently  positive  contributions  of  EDP  to 
the  parent  organizations  were  contradicted  by  reports  of 
negative  effects.  The  Aeroject-General  Corporation  elimi- 
nated all  computer-based  management  information  systems  for 
_heif  company  in  1969.  The  President  of  Aeroject  stated: 

The  computer-based  management  system  is  not  worth  a 
damn. . . By  eliminating  the  system  we  eliminated  a lot  of 
money  from  overhead:  more  importantly  our  management 
capability  improved  dramatically.  We  have  better 
management  at  a lot  lower  cost  (Constant,  1973:2). 

Peat,  Marwick,  Mitchell,  and  Company  (1976)  surveyed 
147  computer  users  and  found  that  less  ,than  one  third  of  the 
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users  were  satisfied  with  the  return  on  their  data  pro- 
cessing investment;  yet,  more  than  90  percent  of  the  users 
expected  the  EDP  resources  to  be  more  important  than  ever  in 
the  next  5 years  in  terms  of  the  organization's  performance 
or  profitability.  In  contrast,  Adams  (1975)  surveyed  75 
managers  and  found  that  most  were  satisfied  with  their 
computer-based  systems. 

These  limited  examples  indicate  that  the  effect  of  EDP 
services  on  an  organization's  overall  performance  was  not 
unidirectional.  Current  and  past  efforts  to  evaluate  the 
EDP  contribution,  usually  reported  under  the  nomenclature  of 
"computer  performance  evaluation",  have  concentrated  almost 
entirely  on  measures  of  efficiency  rather  than  effectiveness 
(Bell,  1971;  Hughes  and  Moe,  1973;  Hellerman  and  Conroy, 
1975)  . Efficiency  measures  generally  take  the  form  of 
hardware  utilization  data  with  the  objective  of  attaining 
greater  utilization  of  the  existing  hardware  and  software 
configuration.  These  measures  are  usually  statistics  such 
as  central  processor  unit  (CPU)  direct-time  utilization, 
core  utilization,  percent  channel  "busy  time",  turnaround 
time,  mix  profiles,  etc.  (Boehm,  1970;  Watson,  1971). 

The  dominance  of  efficiency  measures  has  created 
pressures  that  have  had  some  adverse  effects.  Since  these 
measures  have  traditionally  formed  the  basis  of  evaluation, 
efforts  to  drive  utilization  toward  saturation  levels  have 
naturally  evolved  (Constant,  1973).  McLean  clarified  the 


shortcomings  of  measuring  only  efficiency: 

Efficiency,  as  defined  through  long  usage  in  the 
field  of  engineering,  is  a measure  of  a unit  of  output 
in  terms  of  a unit  of  input.  More  particularly,  in 
computing  it  has  been  defined  as  "the  ratio  of  time 
spent  on  useful  work  to  total  power-on  time".  In  this 
sense,  "useful  work"  means  the  time  during  which  the 
computer  is  not  idle  or  used  for  reruns,  maintenance,  or 
other  non-productive  activity.  Whether  in  fact  this 
computer  work  is  really  useful  - from  a managerial 
standpoint  - is  quite  another  matter.  This  is  much  more 
difficult  to  judge  but  is  a far  more  critical  issue 
(McLean,  1973:95). 

This  "critical  issue"  was  the  impetus  for  this  research 
study.  The  need  for  quantitative  measures  of  effectiveness 
was  often  articulated  without  direction  for  achieving  such 
measures  (Argyris,  1971;  Ward,  1973;  MacWilliams,  1975).  An 
exception  to  this  lack  of  direction  was  provided  by  Lucas 
(1973b)  who  hypothesized  a descriptive  model  of  information 
systems  in  the  context  of  the  organization.  He  proposed 
several  variables  of  interest  as  measures  of  effectiveness. 
His  model,  howeve  suffered  the  same  lack  of  measurement 
instruments  that  racterized  other  empirical  studies  of 
management  infoi  .on  systems.  These  additional  studies 
were  reviewed  by  an  Horn  (1973). 

Although  the  problem  of  effectiveness  measures  could  be 
viewed  from  several  perspectives,  it  seemed  most  appropriate 
to  consider  it  from  the  viewpoint  of  the  user  of  the 
computer-based  information  products  and  services  within  the 
organization.  One  variable  that  integrated  the  user's 
criteria  of  effectiveness  was  the  user's  satisfaction  with 
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the  EDP  support  rendered.  A series  of  management  workshops 
on  EDP  productivity  conducted  at  Ohio  State  University 
consistently  identified  user  satisfaction  as  the  foremost 
indicator  of  effectiveness  (Morris,  1976).  Holmes  (1970:32) 
stated  that  "the  degree  of  user  satisfaction  is  probably  the 
most  useful  index  of  measuring  the  success  or  effectiveness 
of  an  information  system". 

The  importance  of  user  satisfaction  carries  beyond 
merely  a state  of  happiness  or  unhappiness.  Lucas  summa- 
rized some  of  the  effects  in  the  following  statement: 

Information  services  department  practices  influence  user 
attitudes  Poor  attitudes  will  reduce  cooperation; 
users  will  not  encourage  the  design  of  new  systems  and 
will  tend  to  sabotage  existing  ones.  Due  to  badly 
designed  systems,  little  or  no  user  input,  and  poor 
attitudes,  systems  will  not  be  used  unless  mandatory 
. . . information  systems  will  have  little  or  no  influence 
on  user  performance.  In  this  situation  information 
systems  have  failed  because  their  contribution  to  the 
organization  is  nonexistent  or  far  below  success  given 
these  conditions,  and  a cycle  of  continued  failure  can 
be  expected  (Lucas,  1973b:35). 

The  use  of  user  satisfaction  as  an  effectiveness 
indicator  was  reported  in  studies  by  Powers  and  Dickson 
(1973),  Lucas  (1973a),  and  Swanson  (197A) ; however,  the 
focus  of  these  studies  was  not  the  measurement  of  user 
satisfaction.  Consequently,  the  "measurement"  of  user 
satisfaction  was  either  by  subjective  interviews  or 
incomplete  questionnaires.  These  studies  are  reviewed  in 
Chapter  3.  The  inability  to  quantitatively  measure  user 


satisfaction  was  regarded  as  a major  shortcoming. 
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great  British  physicist,  William  Thompson  (Lord  Kelvin)  once 
observed: 

When  you  can  measure  what  you  are  speaking  about, 
and  express  it  in  numbers,  you  know  something  about  it; 
but  when  you  cannot  express  it  in  numbers , your  knowl- 
edge is  of  a meager  and  unsatisfactory  kind;  it  may  be 
the  beginning  of  knowledge,  but  you  have  scarcely,  in 
your  thoughts,  advanced  to  the  stage  of  science  (cited 
by  Wagner,  1976:86). 

PROBLEM  STATEMENT 

In  order  to  quantitatively  measure  user  satisfaction, 
the  following  developments  were  needed.  First,  user 
satisfaction  had  to  be  defined,  and  the  underlying  prop- 
ositions embodied  in  the  definition  had  to  be  substantiated. 
Then,  a measurement  methodology  had  to  be  devised  that  was 
compatible  with  the  definition  to  create  the  actual  means  of 
measurement.  Finally,  the  resultant  means  of  measurement 
had  to  be  evaluated  to  establish  that  user  satisfaction 
could  be  measured  with  an  acceptable  degree  of  reliability. 

Many  benefits  were  envisioned  if  user  satisfaction 
could  be  measured  as  an  indicator  of  EDP  effectiveness. 

These  benefits  included  the  following: 

1.  Standards  of  effectiveness  could  be  established. 

2.  Performance  goals  could  be  set,  and  progress  toward 
those  goals  could  be  tracked. 

3.  The  weak  areas  of  user  support  could  be  identified 
for  corrective  action. 
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4.  The  marginal  effects  of  changes  in  service, 
policies,  etc.  could  be  determined. 

5.  The  emphasis  on  increasing  efficiency  could  be 
complemented  and  counterbalanced  by  a corresponding  emphasis 
on  effectiveness 

6.  Differs  vities  could  be  compared  on  a 

more  complete  baM, 

The  ability  to  measure  user  satisfaction  was  also 
considered  to  be  a key  requirement  in  the  development  and 
investigation  of  the  theoretic  aspects  of  EDP  effectiveness. 
As  Ackoff  (1962:216)  observed,  "Measurement,  perhaps  more 
than  any  other  research  activity,  has  been  the  principal 
stimulus  of  progress  in  both  pure  and  applied  science." 
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RESEARCH  OBJECTIVES 

The  research  objectives  for  this  study  were  predicated 
on  three  propositions.  The  first  proposition  was  that  user 
satisfaction  is  an  attitudinal  construct  of  multi- 
dimensionality. A construct  is  an  attribute  that  exists  in 
every  individual  to  some  degree  but  is  not  physically 
tangible.  An  example  of  a construct  is  intelligence.  An 
attitudinal  construct  is  characterized  by  both  direction 
(positive  or  negative)  and  intensity  of  feeling.  Multi- 
dimensionality implies  that  a construct  is  influenced  by 


• more  than  one  factor. 
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The  second  proposition  was  that  attitudes  can  be 
measured.  The  third  proposition  was  that  a scaling  method- 
ology commonly  used  to  measure  attitudes,  specifically  the 
semantic  differential  methodology,  could  be  adapted  to 
measure  user  satisfaction.  An  analysis  of  these  underlying 
propositions  is  presented  in  Chapter  2. 

Within  the  framework  of  these  propositions,  the  follow- 
ing research  objectives  were  established  for  this  study: 

1.  The  definition  of  the  user  satisfaction  construct 
and  the  formulation  of  a measurement  model  which  reflects 
the  definition. 

2.  The  development  and  the  evaluation  of  the  opera- 
tional means  to  measure  user  satisfaction. 


ORGANIZATION  OF  THE  DISSERTATION 
Chapter  2 presents  an  analysis  of  the  propositions  of 
this  study.  A literature  review  concerning  the  satisfaction 
construct,  measurement  theory,  and  attitude  measurement  is 
reported.  Four  scaling  methodologies  are  evaluated,  and  the 
selection  of  the  semantic  differential  methodology  is 
justified.  The  user  satisfaction  construct  is  defined  and  a 
measurement  model  is  proposed  for  the  construct. 

Chapter  3 presents  a literature  review  pertinent  to  the 
identification  of  factors  important  to  the  user  satisfaction 
construct.  Operational  definitions  are  formulated  for  each 


factor.  An  empirical  test  of  the  factor  list  for  complete- 
ness is  described.  The  test  is  based  on  interviews  with  32 
middle  management  users  in  8 organizations . 


t 

Chapter  4 describes  the  construction  of  the  measurement 
instrument.  The  construction  applies  the  semantic  differ- 
ential methodology  to  the  construct  factors.  Decisions  of 
t 

format  and  a pretest  of  the  instrument  are  also  addressed. 

Chapter  5 presents  an  evaluation  of  the  measurement 
instrument  for  reliability  and  validity.  The  administration 
of  the  instrument  to  29  subjects  and  the  subsequent  analysis 
are  reported.  Scoring  procedures  for  the  instrument  are 
described. 

Chapter  6 presents  the  summary,  conclusions,  and 
recommendations  for  further  research.  This  chapter  con- 
cludes the  dissertation. 


Chapter  2 
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ANALYSIS  OF  UNDERLYING  PROPOSITIONS 

Three  major  propositions  are  central  to  the  achievement 
of  the  research  objectives.  The  propositions  are: 

1.  User  satisfaction  is  a multi-dimentional,  attitu- 
dinal  construct. 

2.  Attitudes  can  be  measured. 

3.  The  semantic  differential  methodology  can  be 
adapted  to  measure  user  satisfaction. 

This  chapter  first  examines  the  general  satisfaction 
construct  as  studied  by  other  investigators  in  organiza- 
tional settings.  The  theoretical  and  empirical  evidence 
examined  in  this  review  shows  the  user  satisfaction  con- 
struct to  be  a multi-dimensional  attitude.  User  satis- 
faction is  defined  as  the  sum  of  feelings  or  affective 
responses  to  distinguishable  factors  of  the  computer- 
based  information  products  and  services  that  are  pro- 
vided within  the  organization.  A measurement  model  is 
formulated  to  reflect  this  definition. 

The  second  portion  of  this  chapter  centers  on  the 
measurement  theory  that  is  germane  to  attitude  measurement. 
The  concept  of  attitude  is  examined  to  establish  the  nature, 
components,  and  properties  of  the  concept  to  facilitate  the 
measurement  process.  Four  traditional  scaling  methodologies 
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for  measuring  attitudes  are  reviewed.  This  review  culmi- 
nates in  the  selection  of  the  semantic  differential  for 
adaptation  to  measure  the  user  satisfaction  construct. 

The  final  portion  of  this  chapter  focuses  on  a compre- 
hensive evaluation  of  the  semantic  differential  methodology. 
This  evaluation  provides  evidence  of  reliability,  validity, 
and  robustness  against  rating  errors  for  confirmation  of  the 
semantic  differential  selection.  Other  advantages  of  the 
methodology  are  also  identified. 

The  conclusions  of  this  chapter  indicate  that  the 
research  objectives  are  feasible.  Furthermore,  the  analysis 
of  the  fundamental  propositions  provides  an  answer  of  "how" 
to  measure  user  satisfaction. 

THE  SATISFACTION  CONSTRUCT 
The  concept  of  satisfaction,  particularly  in  the 
organizational  environment,  has  been  a major  subject  of 
study  by  psychologists  (Wanous  and  Lawler,  1972) . The  bulk 
of  these  studies  was  directed  at  a set  of  satisfaction 
objects  collectively  referred  to  as  "job  satisfaction" 
(Smith,  Kendall,  and  Hulin,  1969;  Schwab  and  Cummings, 

1973).  The  research  concerned  with  job  satisfaction  was 
reviewed  to  develop  a basic  understanding  of  the  character 
of  the  satisfaction  construct  in  organizational  settings. 

Definitional  shortcomings  were  cited  as  a major 
research  and  theoretical  impediment  by  Strong  (1967)  and  by 
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Schwab  and  Cummings  (1973).  Strong  (1967:680)  stated,  "What 
is  satisfaction?  Some  say  it  is  a kind  of  feeling  as  simple 
as  pleasantness;  others  contend  it  is  a complex  of  feeling, 
emotion,  and  sensation."  Subsequent  evaluation  of  satis- 
faction definitions  identified  two  general  categories. 

These  were  expressed  by  Schwab  and  Cummings  after  an  inten- 
sive literature  review: 

First,  it  is  often  unclear  whether  satisfaction  is 
being  used  in  a "narrow",  need  deprivation  sense,  or  in 
a "broad",  attitudinal  sense. . . Beginning  with  the  need 
deprivation  versus  attitude  issue,  satisfaction- 
dissatisfaction  may  be  thought  of  in  the  context  of 
"elementary"  motivation  theory.  Needs,  demands,  or 
drives  generate  tensions  (feelings  of  dissatisfaction) . 
The  individual  engages  in  behavior  designed  to  obtain 
goals  or  incentives  to  reduce  the  tensions  (satisfy  the 
need) . 

Alternatively,  satisfaction-dissatisfaction  can  be 
thought  of  as  the  evaluative  component  of  an  attitude. 

A person  may  respond  affectively  (feel  satisfied  or 
dissatisfied)  about  an  object  or  referent  in  his  work 
environment  (Schwab  and  Cummings,  1973:137). 

These  two  viewpoints  of  satisfaction  were  recognized  by 
Smith,  Kendall,  and  Hulin  (1969).  They  defined  satisfaction 
from  the  attitudinal  perspective  but  acknowledged  the  effect 
of  the  need-deprivation  influence  on  individual  frames  of 
reference  or  satisfaction  "anchor  points".  Although  the 
need-deprivation  definition  found  considerable  support  in 
theoretical  writings  (March  and  Simon,  1958;  Porter,  1961; 
Locke,  1969),  most  measurement  studies  reported  were  based 
primarily  on  the  attitudinal  definition  (Brayfield  and 
Rothe,  1951;  Smith,  Kendall,  and  Hulin,  1969;  Cross,  1973; 
Churchill,  Ford,  and  Walker,  1974). 
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Wanous  and  Lawler  (1972)  summarized  the  measurement 
models  that  evolved  from  these  definitions.  Although  nine 
models  were  reported,  there  were  two  basic  models.  The 
first  model,  based  on  the  need-deprivation  definition  and 
referred  to  as  the  discrepancy  model,  was  defined  by 

Si  - I dij  - Ai;j)  WAj  (2.1) 

j=l 

where 

Si  = Satisfaction  of  individual  i 
Iij  = Ideal  state  of  factor  j for  individual  i 
Aij  = Actual  state  of  factor  j for  individual  i 
Wij  = Importance  weight  of  factor  j for  individual  i 
The  discrepancy  model  was  based  on  the  hypothesis  that 
satisfaction  feelings  were  associated  with  the  perceived 
difference  between  an  ideal  state  and  the  actual  state  of 
distinguishable  factors  in  relation  to  alternatives  that 
were  available  in  a given  situation.  Thus,  the  feelings 
were  influenced  by  not  only  the  actual  state  of  a particular 
factor  but  also  by  the  internal  standards  maintained  by  an 
individual  under  differing  circumstances. 

Ideally,  a unique  anchor  point  representing  the  ideal 
state  for  each  factor  for  each  individual  subject  was 
required.  The  alternatives  that  an  individual  perceived  as 
available  altered  this  frame  of  reference.  The  problems  of 
establishing  these  anchor  points  for  individuals  was  a major 
impediment  to  implementing  this  model. 
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The  second  model  was  based  on  the  attitudinal  defini- 
tion of  satisfaction.  It  was  defined  by 

Si  - ?w ij  Rij  (2.2) 

where 

= Satisfaction  of  individual  i 
Wj_j  = Importance  weight  of  factor  j for  individual  i 
R^j  = Raw  score  of  factor  j for  individual  i 
In  this  model,  the  raw  score  of  a factor  was  determined 
by  the  position  of  an  individual's  feelings  about  that 
factor  on  a continuum  between  a "worst  state"  and  a "best 
state"  for  the  factor.  The  measurement  problem  was  there- 
fore changed  from  the  determination  of  an  ideal  state  con- 
trasted to  an  actual  state  to  the  placement  of  an 
individual's  feelings  on  a common  continuum.  Anchor  points 
acceptable  to  a wide  variety  of  individuals  were  still 
required  to  establish  the  underlying  continuum. 

The  other  measurement  models  reported  by  Wanous  and 
Lawler  (1972)  were  primarily  variations  of  these  two  basic 
models.  One  common  variation  was  achieved  by  assigning 
factor  weights  of  unity  which  resulted  in  unweighted 
summation  models. 

All  of  the  measurement  models  recognized  a basic 
characteristic  of  the  satisfaction  construct.  The  global  or 
overall  satisfaction  score  was  composed  of  independent 
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scores  assigned  to  the  distinguishable  factors.  Cross 
stated: 


Job  satisfaction  is  still  commonly  referred  to  as  if  it 
were  a single  variable,  but  this  is  unquestionably  too 
simplistic,  and  the  trend  is  to  consider  it  as  a multi- 
dimensional construct.  The  reasons  for  this  are  largely 
self-evident. . . it  is  feasible  to  imagine  situations 
where  groups  of  workers  are  indiscriminable  in  terms  of 
their  scores  on  an'  overall  measure,  yet  differ  widely  in 
terms  of  their  satisfaction  with  different  aspects  of 
their  employment.  It  seems  likely,  therefore,  that  the 
studies  of  separate  dimensions  of  job  satisfaction  will 
be  more  fruitful  than  research  which  employs  only  a 
single  global  measure  (Cross,  1973:193). 

Evans  had  argued  the  same  viewpoint  earlier: 

The  concept  of  job  satisfaction  is  a many-faceted 
one.  Although  some  students  see  it  as  a generalized 
affective  orientation  to  all  aspects  of  the  work 
situation,  it  is  clear  that  such  a view  expresses  the 
resultant  of  a whole  host  of  orientations  to  specific 
aspects  of  the  job  (Evans,  1969:93). 

This  multi-dimensional  character  of  job  satisfaction 
created  two  measurement  concerns.  One  was  the  specification 
of  the  facets  or  factors  that  were  relevant  in  a given 
setting.  The  second  was  the  consideration  of  differential 
weights  to  reflect  the  importance  of  the  facets.  Notwith- 
standing these  measurement  concerns,  Wanous  and  Lawler 
(1972:103)  concluded  that  "the  evidence  from  the  convergent 
and  discriminant  validity  matrix  suggests  that  it  is 
possible  to  validly  measure  peoples’  satisfaction  with 
different  facets  of  their  job." 

This  conclusion  was  supported  by  the  wide  acceptance  of 
standardized  instruments  which  measure  multiple  facets  of 
job  satisfaction.  The  Science  Research  Associates  (SRA) 
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Employee  Inventory  measured  the  attitudinal  feelings  of 
individuals  and  groups  toward  14  facets  of  job  satisfaction 
(Miller,  1970).  Moore  and  Burns  (1956)  reported  that  this 
instrument  had  been  administered  to  over  500,000  subjects. 
Another  standardized  instrument,  the  Job  Description  Index, 
measured  satisfaction  feelings  toward  five  facets  of  the  job 
environment  (Smith,  Kendall,  and  Hulin,  1969).  Both  of 
these  measurement  instruments  were  supported  by  evidence  of 
reliability  and  validity. 

The  review  of  job  satisfaction  literature  supported  the 
following  conclusions  about  the  satisfaction  construct  in 
organizational  settings: 

1.  Satisfaction  could  be  defined  as  an  attitude. 

2.  Satisfaction  was  a multi- dimensional  construct. 

3.  Global  satisfaction  measures  were  composed  of 
individual  satisfaction  feelings  toward  the  distinguishable 
factors  that  comprised  the  construct  domain. 

4.  Satisfaction  could  be  measured. 

The  extension  of  these  conclusions  from  an  individual's 
job  satisfaction  to  a user's  satisfaction  with  the  computer- 
based  information  products  and  services  that  are  provided 
within  the  organization  was  straightforward.  Powers  and 
Dickson  (1973:150)  defined  user  satisfaction  as  the  "atti- 
udes  of  managers  receiving  project  products  relative  to  how 
well  their  information  needs  were  being  satisfied."  This 


i 


definition  of  user  satisfaction  as  an  attitude  was  also 


supported  by  Lucas  (1973b),  Swanson  (1974),  and  Adams 
(1975). 

The  multi-dimensional  character  of  user  satisfaction 
was  also  amply  substantiated.  Firnberg  (1973)  identified  16 
different  factors  that  influenced  user  satisfaction  in  two 
case  studies  of  different  organizations.  He  indicated  that 
the  factors  contributed  differentially  to  the  overall  global 
feeling  of  satisfaction.  Other  writers  who  identified 
multiple  factors  affecting  user  satisfaction  included  Fitts 
(1971),  Stone  and  Tarnowieski  (1972),  Colton  (1973), 

Constant  (1973),  and  Schussel  (1974). 

Therefore,  user  satisfaction  is  defined  as  the  sum  of 
feelings  or  affective  responses  to  distinguishable  factors 
of  the  computer-based  information  products  and  services 
that  are  provided  within  the  organization.  The  measurement 
model  which  reflects  this  definition  is  specified  by 
Equation  2.2. 

Si  - ? ■ Wia  Rij 

Application  of  this  model  imposes  two  very  different 
requirements.  One  is  the  delineation  of  the  factors  com- 
prising the  domain  of  the  user  satisfaction  construct.  The 
second  is  the  measurement  of  attitudinal  feelings  toward 
those  factors.  The  following  sections  of  this  chapter 
review  measurement  theory,  attitudes,  and  attitude  measure- 
ment for  insight  to  achieve  the  measurement  objective. 
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Scaling  methodologies  are  then  evaluated  to  select  a method 


of  measuring  the  user  satisfaction  feelings.  The  domain 
specification  is  developed  in  Chapter  3. 


MEASUREMENT 

In  a discussion  of  the  fundamental  nature  of  measure- 
ment, Lorge  observed: 

"Measure"  is  one  of  the  thousand  most  common  words 
in  printed  English.  As  is  usual  with  words  that  have 
had  a long  history  and  wide  currency,  "measure"  has  many 
different  meanings  and  applications.  In  a count  of  its 
occurrence  in  a sample  of  two  and  a half  million  words, 
"measure”  occurred  more  than  four  hundred  times  and  was 
used  in  forty  different  ways  (Lorge,  1967:43). 

This  diversity  of  usage  sparked  an  active  debate  on 
definitions  between  scientists,  philosophers , and  others; 
however,  the  definition  of  measurement  often  repeated  in  one 
form  or  another  was  expounded  by  Stevens: 

We  may  say  that  measurement,  in  the  broadest  sense, 
is  defined  as  the  assignment  of  numbers  to  objects  or 
events  according  to  rules.  The  fact  that  numerals  can 
be  assigned  under  different  rules  leads  to  different 
kinds  of  scales  and  different  kinds  of  measurement.  The 
problem  then  becomes  that  of  making  explicit  (a)  the 
various  rules  for  the  assignment  of  numerals,  (b)  the 
mathematical  properties  (or  group  structure)  of  the 
resulting  scales,  and  (c)  the  statistical  operations 
applicable  to  measurements  made  with  each  type  of  scale 
(Stevens , 1946 : 677) . 

This  assignment,  or  classification,  of  numbers  to 
objects  or  events  was  almost  invariably  performed  with  some 
purpose  in  mind.  As  Caws  succinctly  stated,  "Measurement 


presupposes  something  to  be  measured,  and  unless  we  know 
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what  that  something  is,  no  measurement  can  have  any  signi- 
ficance" (cited  by  Churchman  and  Ratoosh,  1959:1).  The 
philosopher  and  mathematician  Russell  echoed  that  sentiment, 
"Measurement  demands  some  one-to-one  relation  between  the 
numbers  and  magnitudes  in  question  - a relation  which  may  be 
direct  or  indirect,  important  or  trivial,  according  to 
circumstances"  (cited  by  Churchman  and  Ratoosh,  1959:21). 

These  statements  indicate  that  measurement  is  always 
concerned  with  some  feature  or  attribute  of  the  subjects  of 
interest.  Attributes  which  are  not  concrete  and  tangible 
are  labeled  "constructs"  because  they  are  fabricated  intel- 
lectually rather  than  conceived  as  physical  entities 
(Ghiselli,  1964:10).  A process  of  abstraction  is  necessary 
to  define  and  to  describe  any  construct  before  measurement 
of  "how  much"  of  the  attribute  is  present  in  the  measurement 
object.  Cassier  made  the  point  that: 

All  measure  has  to  be  "conceived  and  sought"  before  it 
can  be  found  in  experience;  i.e.,  one  has  first  a 
concept  of  some  quality  and  looks  afterwards  for 
quantitative  expressions  of  it  (cited  by  Churchman  and 
Ratoosh,  1959:8). 

Yet,  conception  and  abstraction  of  constructs  such  as 
attitudes  satisfy  only  the  first  requirement  of  Stevens' 
(1946)  definition  of  measurement.  Rules  must  be  formulated 
to  assign  numerals  to  achieve  certain  mathematical  prop- 
erties. Although,  many  different  rule  sets  have  been 
utilized  which  result  in  correspondingly  different  scales, 
four  basic  scales  have  evolved  into  common  usage  and 


terminology.  These  are  summarized  in  Table  2.1. 


The  nominal  scale  represents  the  most  unrestricted 
assignment  of  numbers.  The  numbers  represent  categories  of 
objects.  The  information  that  may  be  derived  from  this 
scale  is  confined  to  operations  which  depend  solely  on 
frequencies,  such  as  the  mode  (the  most  numerous  class). 
Certain  contingency  tests  may  be  performed  that  are  based  on 
the  distribution  of  objects  among  the  different  classes  or 
categories . 

The  ordinal  scale  is  derived  by  a rank-ordering  of 
objects  according  to  the  attribute  of  interest.  This 
operation  conveys  only  the  information  about  the  relative 
positions  of  the  objects.  There  is  no  indication  of  "how 
much"  of  the  attribute  is  present  or  how  far  apart  the 
positions  are  with  respect  to  the  attribute.  This  short- 
coming precludes  the  use  of  some  of  the  more  powerful 
methods  of  mathematics  because  the  mean  and  standard 
deviation  cannot  be  used  in  the  analysis. 

The  interval  scale  not  only  rank-orders  the  objects  but 
also  expresses  how  far  apart  the  objects  are  with  respect  to 
the  attribute.  It  does  not  convey  information  about  the 
"absolute"  magnitude  of  the  attribute.  The  usual  statis- 
tical measures  of  means,  standard  deviatons , correlations, 
and  others  may  be  derived  from  interval  scales . The  broad 
score  of  the  analytical  methods  which  may  be  employed 
increase  the  desirability  of  achieving  an  interval  scale  in 
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Common  Measurement  Scales 


Scale 

Basic  Empirical 
Operations 

Mathematical 

Group  Structure 

Nominal 

Determination  of 
equality 

Permutation  group 
x'  = f(x) 
where  f(x)  means 
any  one-to-one 
substitution 

Ordinal 

Determination  of 
greater  or  less 

Isotonic  group 
x'  = f(x) 
where  f(x)  means 
any  mono tonic 
increasing  function 

Interval 

Determination  of 
equality  of  intervals 
or  differences 

General  linear  group 
x'  = ax  + b 

Ratio 

Determination  of 
equality  of  ratios 

Similarity  group 
x ' = ax 
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the  measurement  process. 

The  ratio  scale  has  all  of  the  properties  of  the  inter- 
val scale  plus  the  existence  of  rational  zero  point  with  a 
known  distance  of  the  intervals  from  the  zero  point.  These 
properties  allow  multiplicative  transformations.  The  ratio 
scale  conveys  information  about  the  absolute  magnitude  of 
the  attribute  "present"  in  an  object. 

The  scale  characteristics  of  a particular  measure  of 
attitude  must  be  judged  in  an  inferential  manner.  There  are 
no  ostensive  characteristics  of  psychological  attributes 
that  can  be  correlated  reasonably  to  specific  scale  prop- 
erties, such  as  a rational  zero  point.  There  are  perhaps 
trivial  examples,  such  as  the  amount  of  "intelligence"  in  a 
stone  or  a deceased  person,  but  these  are  meaningless  in 
consideration  of  significant  measurement  problems. 

Nunnally  (1967)  presented  a cogent  argument  that  each 
scaling  model  constitutes  a set  of  axioms  or  assumptions 
concerning  how  the  data  should  look  when  the  measure  is  put 
to  use.  He  stated: 

If  the  data  obtained  from  applying  a measurement  scale 
fit  the  axioms  of  the  model  under  consideration  and  the 
axioms  (assumptions)  of  the  model  are  correct,  the 
measure  has  scale  properties  specified  by  the  model 
(Nunnally,  1967:23). 

This  method  of  inferring  the  scale  characteristics  from 
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the  assumptions  and  the  resulting  empirical  data  is  an 
attempt  to  test  the  fit  to  a model.  The  realities  of  meas- 
uring attitudes  or  other  psychological  phenomena  suggest 
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that  this  is  appropriate.  Scales  of  attributes  that  are 

accepted  as  "good"  measures  of  certain  attributes  eventually 

become  established  as  conventions.  Nunnally  concluded: 

Most  persons  will  agree  that  measurement  plans  based  on 
common  sense  and  past  experience  improve  the  probabil- 
ities of  finding  a measurement  scale  that  eventually 
will  be  accepted  as  a useful  convention. 

Within  this  definitional  framework  of  measurement  and 
the  resultant  scale  properties,  the  writings  of  prominent 
social  psychologists  are  reviewed  to  provide  insight  into 
the  nature,  components,  and  properties  of  the  concept  of 
attitude.  This  review  is  followed  by  an  evaluation  of 
specific  scaling  methodologies  developed  to  measure 
attitudes . 


THE  CONCEPT  OF  ATTITUDE 

After  an  extensive  review  of  the  history  of  the  concept 
of  attitude,  Allport  (1967:3)  concluded  that  "the  concept  of 
attitude  is  probably  the  most  distinctive  and  indispensable 
concept  in  contempory  American  social  psychology."  He  then 
synthesized  a multitude  of  definitions  into  a definition  of 
attitude  that  was  widely  referenced: 

An  at  itude  is  a mental  and  neural  state  of  readi- 
ness, organized  through  experience,  exerting  a directive 
or  dynamic  influence  upon  the  individual's  response  to 
all  objects  and  situations  with  which  it  is  related 
(Allport,  1967:8). 

Allport's  (1967)  viewpoint  that  attitudes  were  bipolar 
in  direction  (positive  or  negative,  favorable  or  unfavor- 
able) was  shared  by  Thurstone  (1967)  and  Shaw  and  Wright 
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(1967)  among  others.  This  led  to  attitude  being  conceptu- 
alized as  a unidimensional  concept. 

Doob  (1967)  challenged  Allport's  definition  and  argued 

that  behavior  and  attitude  were  not  necessarily  related. 

Chein  responded  to  Doob's  argument  with  the  proposal  that 

attitude  was  actually  composed  of  three  elements: 

...  an  affective  or  feeling  aspect,  a cognitive  aspect 
representing  an  individual's  evaluation  of  or  beliefs 
about  an  attitude  object,  and  a conative  aspect 
representing  an  action  tendency  toward  the  attitude 
object  (cited  by  Summey,  1974:24). 

The  multiple  component  definition  of  attitude,  although 
often  included  in  theoretical  discussions,  was  rarely  util- 
ized in  attitude  measurement  studies.  Measurements  of 
attitudes  regularly  reverted  to  the  unidimensional  approach 
that  considered  the  evaluative  component  to  be  the  essence 
of  the  attitude  concept.  Shaw  and  Wright  observed: 

In  short,  the  typical  attitude  scale  measures  the 
acceptance  of  evaluative  statements  about  the  attitude 
object.  The  attitude  toward  the  object  is  inferred  from 
the  statements  endorsed  by  the  subject,  based  upon  the 
consentual  evaluation  of  the  nature  of  the  character- 
istics attributed  to  the  object  by  the  acceptance  of 
these  statements  (Shaw  and  Wright,  1967:14). 

This  viewpoint  was  expressed  earlier  in  Thurstone's 
definition  of  attitude  as: 

. . . the  intensity  of  positive  or  negative  affect  for  or 
against  a psychological  object.  A psychological  object 
is  any  symbol,  person,  phrase,  slogan,  or  idea  toward 
which  people  can  differ  as  regards  positive  or  negative 
affect  (Thurstone,  1931:250). 

This  definition  was  adopted  by  many  researchers, 
including  Osgood,  Suci,  and  Tannenbaum  (1957),  Anderson  and 
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Fishbein  (1965) , and  Shaw  and  Wright  (1967) . The  definition 
expressed  two  characteristics  of  attitude  that  had  important 
implications  for  the  measurement  of  attitude.  These  were 
direction  and  intensity  of  feeling.  Thus,  the  concept  of 
attitude  was  subsumed  for  measurement  purposes  as  a set  of 
evaluative  feelings  with  both  direction  and  intensity  toward 
the  characteristics  of  an  attitude  object  (Oppenheim,  1966). 

This  concept  paralleled  the  definition  of  user  satis- 
faction and  was  mirrored  in  the  measurement  model  proposed 
in  Equation  2.2.  It  was  concluded  that  the  conceptual 
differences  between  attitude  and  user  satisfaction  were 
insignificant  in  terms  of  the  measurement  purpose.  Smith, 
Kendall,  and  Hulin  (1969:1)  stated,  "The  problems  associated 
with  the  measurement  of  satisfaction  are  but  specific 
examples  of  those  encountered  in  the  measurement  of  any 
attitude."  Therefore,  the  scaling  methodologies  that  were 
traditionally  used  to  measure  attitudes  were  evaluated  for 
potential  application  in  measuring  user  satisfaction. 

ATTITUDE  SCALING  METHODOLOGIES 

The  most  frequently  used  methods  of  measuring  attitudes 
were  developed  by  Thurstone,  Likert,  and  Guttman  (Shaw  and 
Wright,  1967;  Nunnally,  1967).  A popular  supplement  to 
these  methods  was  the  semantic  differential  technique 
developed  in  the  1950's  by  Osgood,  Suci,  and  Tannenbaum 
(1957) . Each  of  these  methodologies  was  evaluated  for 
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possible  adaptation  to  the  purpose  of  measuring  user 
satisfaction.  The  methodology  descriptions  were  condensed 
from  writers  that  concentrated  on  attitude  measurement 
(Torgerson,  1958;  Shaw  and  Wright,  1967;  Nunnally,  1967; 
Miller,  1970). 


Thurstone  Equal -Appearing  Interval  Scale 

This  method  of  scale  construction  requires  a large 
number  of  statements  (items)  related  to  the  attitude  object. 
The  items  are  considered  nonmonotone.  This  implies  that 
persons  above  or  below  (positive  or  negative  direction, 
respectively)  the  attitude  position  reflected  by  an  item  may 
disagree  with  the  item.  The  steps  involved  in  constructing 
the  scale  are  as  follows  (Shaw  and  Wright,  1967:21;  Miller, 
1970:91) : 

1.  Several  hundred  statements  related  to  the  attitude 
object  being  investigated  are  formulated. 

2.  A large  number  of  judges  (50  - 300)  independently 
classify  the  statements  into  eleven  groups  ranging  from  the 
most  favorable  to  neutral  to  least  favorable.  These  groups 
are  numbered  1 to  11. 

3.  The  scale  value  of  an  item  is  computed  as  the 
median  position  to  which  it  is  assigned  by  the  group  of 
judges . 

4.  Statements  which  have  too  broad  a spread  are 
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discarded  as  ambiguous  or  irrelevant. 


5.  The  scale  is  formed  by  selection  of  items  which  are 
evenly  spread  along  the  scale  from  one  extreme  to  the  other. 

The  resulting  group  of  statements,  each  with  an  assign- 
ed scale  value,  can  then  be  presented  to  a subject  with 
instructions  to  check  those  statements  with  which  the 
subject  agrees.  The  subject's  "scope"  is  the  median  value 
of  all  of  the  items  that  are  checked.  Theoretically,  the 
subject  will  agree  with  only  a few  statements  that  are 
clustered  near  his  "true"  attitude  position.  An  example  of 
Thurstone  type  scale  items  is  shown  in  Table  2.2. 


Table  2.2* 

Example  of  Thurstone  Type  Scale  Items 


My  job  is  like  a hobby  to  me. 

I am  satisfied  with  my  job  for  the  time  being. 

I am  often  bored  with  my  job. 

Most  of  the  time  I have  to  force  myself  to  go  to  work. 


* Adapted  from  Miller  (1970:92) 


This  method  of  scale  construction  was  extensively 
utilized  to  measure  various  attitudes  toward  referent 
objects  or  concepts,  including  job  satisfaction,  war, 
church,  capital  punishment,  and  others  (Miller,  1970).  The 
validity  of  the  scales  depended  largely  upon  the  skill  of 
the  investigator  formulating  the  statements  related  to  the 
attitude.  Nunnally  (1967)  argued  that  it  was  extremely 
difficult  to  achieve  items  that  have  nonmonotone  charac- 
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teristics.  This  led  to  many  of  the  statement  items, 
particularly  those  related  to  the  middle  of  the  scale 
values,  to  be  populated  with  and's,  but's,  or's,  and  other 
"double-barrelled"  indicators  within  statements  (Nunnally, 
1967:530) . 

The  primary  advantage  of  the  Thurstone  type  scales  was 
that  attitudes  could  be  interpreted  directly  from  the 
individual  scores  without  reverting  to  normalized  scores 
which  were  required  in  most  summation  type  scales.  Nunnally 
(1967:530)  criticized  the  scales,  however,  as  being  of 
limited  use  and  precision  when  comparing  differences  in 
attitudes  of  individuals  and  groups  of  people. 

There  were  disadvantages  to  using  this  methodology  for 
measuring  user  satisfaction.  There  were  multiple  factors 
which  were  thought  to  affect  user  satisfaction.  The  multi- 
dimensional character  of  the  user  satisfaction  construct 
indicated  that  richer  information  would  be  provided  if  each 
factor  could  be  measured  independently.  A vast  number  of 
statements,  however,  would  be  required  to  adequately  cover 
the  continuum  underlying  each  factor.  Therefore,  the 
Thurstone  equal-appearing  interval  method  was  not  considered 
to  be  an  economical  means  to  the  measurement  purposes . 


Likert-Type  Scale 


The  Likert  method  of  scale  construction,  like  the 
Thurstone  method,  requires  a large  number  of  statements 


about  the  attitude  being  investigated.  The  statements  are 
monotone  items  with  the  characteristic  that  the  intensity  of 
the  subject's  agreement  or  disagreement  with  the  item 
results  in  a higher  or  lower  score  for  that  item.  The  steps 
involved  in  constructing  the  scale  are  as  follows  (Shaw  and 
Wright,  1967:24;  Miller,  1970:92): 

1.  A large  number  of  clearly  favorable  or  clearly 
unfavorable  statements  about  attitude  are  formulated. 

2.  These  items  are  administered  to  a representative 
group  of  subjects.  They  are  asked  to  respond  to  each 
statement  by  indicating  their  reaction  as: 

a.  Stongly  agree 

b.  Agree 

c.  Undecided 

d.  Disagree 

e.  Strongly  disagree 

3.  The  responses  are  scored  as  5,  4,  3,  2,  1,  respec- 
tively. Negatively  worded  statements  are  scored  in  reverse 
order,  so  that  a high  score  indicates  the  most  favorable 
attitude. 

4.  Each  individual  subject's  total  score  is  computed 
by  the  addition  of  the  item  scores. 

5.  The  responses  are  analyzed  to  determine  which  items 
differentiate  most  clearly  between  the  highest  and  lowest 
quartiles  of  total  scores. 
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6.  The  items  which  differentiate  best  are  used  to  form 
a scale. 

The  Likert-type  scale  which  results  from  the  above 
procedures  is  generally  considered  an  ordinal  scale  which 
orders  individuals  along  the  attitude  continuum;  therefore, 
interpretation  of  the  scores  has  meaning  only  within  the 
distribution  of  the  scores  within  the  sample.  This  implies 
that  the  scale  scores  should  be  standardized  on  a sample 
drawn  from  the  subject  population  of  interest.  These 
standardized  scores  provide  the  "yardstick"  for  interpre- 
tation of  future  subject  scores. 

An  example  of  the  Likert  type  scale  is  presented  in 
Table  2.3.  The  subject  is  required  to  indicate  both  the 
direction  (agreement  or  disagreement)  and  intensity  of  his 
feeling  about  the  statement. 


Table  2.3* 

Example  of  Likert  Type  Scale  Items 


(1)  The  future  looks  very  black. 


Strongly 
Agree  Agree 


Strongly 

Uncertain  Disagree  Disagree 


(2)  Most  people  can  be  trusted. 


Strongly 
Agree  Agree 


Strongly 

Uncertain  Disagree  Disagree 


* Adapted  from  Miller  (1970:93). 
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Development  of  a Likert-type  scale  was  generally 
considered  more  economical  than  the  development  of  a 
Thurstone  equal-appearing  interval  scale,  but  the  problem  of 
the  construct  domain  representation  by  the  set  of  statements 
was  common  to  both.  Moreover,  the  ordinal  property  of  the 
Likert  scale  was  a severe  disadvantage.  The  satisfaction 
model  presented  in  Equation  2.2  was  based  on  the  formulation 
of  anchor  points  for  each  factor.  The  Likert- type  scales 
were  not  necessarily  based  upon  anchor  points  which  applied 
to  a comprehensive  user  population;  therefore,  this  scaling 
methodology  was  not  considered  further. 

Guttman  Scalogram  Analysis 

This  method  of  scale  construction  is  based  on  a set  of 
monotone  statements  that  vary  in  favorableness  or  unfavor- 
ableness about  the  attitude  being  investigated.  It  is 
predicated  on  the  theory  that  a unidimensional  attitude 
continuum  results  in  a predicted  pattern  of  responses.  An 
individual  responding  positively  to  one  statement  is 
expected  to  respond  positively  to  all  other  statements  with 
a lower  degree  of  favorableness.  Thus,  the  intensity  of  the 
attitude  is  contained  within  the  arrangement  of  the  attitude 
statements  rather  than  in  the  subject's  response,  as  in  the 
Likert- type  scales. 

The  resulting  pattern  of  responses  is  evaluated  to 
determine  if  the  statements  are  scalable  (unidimensional)  or 
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not.  The  pattern  is  determined  by  arranging  the  statements 
in  order  of  lowest  to  highest  favorableness  and  listing  the 
responses  of  the  individual  subjects  in  decreasing  order  of 
the  total  number  of  positive  responses.  An  example  is 
illustrated  in  Table  2.4.  A positive  response  is  indicated 
by  a "+".  The  statement  items  are  ordered  by  the  number  of 
positive  responses  received  from  all  of  the  subjects.  The 
subject  scores  reflect  the  number  of  positive  responses  to 
the  six  items. 


Table  2.4 

Example  of  Guttman  Scalogram 


Least 

Host 

Favorable 

Statements 

Favorable 

Subject 

3 

6 

1 

4 

2 

5 

Score 

5 

+ 

+ 

+ 

— 

+ 

+ 

5 

3 

+ 

+ 

+ 

+ 

4 

1 

+ 

+ 

+ 

+ 

4 

2 

+ 

- 

+ 

+ 

3 

4 

+ 

+ 

2 

The 

reproducibility  of 

the 

scale  is 

determined  by 

the 

deviations  (errors)  from  the  expected  response  patterns. 
The  errors  in  Table  2.4  are  indicated  by  a The 

coefficient  of  reproducibility  is  calculated  by 


R = 1 - total  number  of  errors 

total  number  of  responses 
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If  the  coefficient  of  reproducibility  is  greater  than 
.90,  the  statements  are  considered  scalable.  The  steps  in 
constructing  the  Guttman-type  scale  are  as  follows  (Shaw  and 
Wright,  1967:25:  Miller,  1970:94): 

1.  Formulate  statements  that  are  applicable  to  the 
attitude  being  investigated. 

2.  Administer  the  statements  to  a sample  population. 

3.  Discard  statements  with  more  than  80  percent 
agreement  or  disagreement. 

4.  Order  respondents  from  most  favorable  responses  to 
fewest  favorable  responses.  Order  from  top  to  bottom. 

5.  Order  statements  from  most  favorable  responses  to 
fewest  favorable  responses.  Order  from  left  to  right. 

6.  Discard  statements  that  fail  to  discriminate 
between  favorable  respondents  and  unfavorable  respondents. 

7.  Calculate  the  coefficient  of  reproducibility.  If 
the  coefficient  is  greater  than  or  equal  to  .90,  a 
unidimensional  scale  is  said  to  exist. 

8.  Score  each  respondent  by  the  number  of  favorable 
responses . 

The  resulting  Guttman  scales  did  not  insure  equal 
intervals  (equality  of  units) , but  rather  resulted  in  an 
ordinal  ordering  of  the  subjects  along  a unidimensional 
attitude  continuum.  The  same  disadvantages  of  domain 
determination  and  anchor  point  descriptions  of  the  Thurstone 
and  Likert  methods  also  applied.  The  Guttman  method  was 
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conveyed  understanding  about  the  meaning  of  things,  it  was 
reasonable  to  assume  that  adjectives  could  be  used  in  a 
rating  process  to  measure  the  meaning  associated  with  the 
objects  or  concepts  of  interest  (Nunnally,  1967). 

The  measurement  process  involved  rating  the  measurement 
object  on  a series  of  bipolar  adjective  scales  separated  by 
a fixed  number  of  intervals  (usually  seven).  The  subject 
was  asked  to  mark  the  position  in  the  interval  which  indi- 
cated both  the  direction  and  the  intensity  of  his  feeling 
toward  the  measurement  object  as  indicated  by  the  polar 
adjective  terms.  An  example  of  bipolar  scales  is  shown  in 
Table  2.5. 
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Table  2.5 


Example  of  Semantic  Differential  Scales 

- — 

» 

good 
active 


t strong 


Supervisors 


: bad 
passive 
: weak 


l 
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Meaning  was  considered  to  exist  as  a semantic  space 

that  was  Euclidian  in  nature.  The  originators  of  the 

semantic  differential  stated: 

The  essential  operation  of  measurement  is  the  successive 
allocation  of  a concept  to  a series  of  descriptive 
scales  defined  by  polar  adjectives,  these  scales  select- 
ed so  as  to  be  representative  of  the  major  dimensions 
along  which  meaningful  processes  vary.  In  order  to 
select  a set  of  scales  having  these  properties,  it  is 
necessary  to  determine  what  the  major  dimensions  of  the 
semantic  space  are  (Osgood,  Suci,  and  Tannenbaum, 
1957:31)  . 

An  impressive  variety  of  studies  was  reported  by 
Osgood,  Suci,  and  Tannenbaum  (1957)  which  utilized  factor 
analysis  to  explore  the  dimensions  of  the  semantic  space. 

The  studies  consistently  identified  three  general  dimensions 
which  were  labeled  Evaluation,  Potency,  and  Activity,  often 
referred  to  as  the  EPA  structure.  Certan  adjective  pairs 
were  identified  with  heavy  loadings  on  these  three  general 
dimensions.  Examples  are  shovm  in  Table  2.6. 
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Table  2.6* 

Sample  EPA  Adjective  Pairs 


Dimensions 


Evaluation 


Potency 


Activity 


Adjective  Pairs 


good  - bad;  fair  - unfair; 

wise  - foolish;  positive  - negative 

strong  - weak;  hard  - soft; 
heavy  - light;  severe  - lenient 

active  - passive;  quick  - slow; 
busy  - lazy;  sharp  - dull 


* Adapted  from  Nunnally  (1967:537) 


Several  writers  equated  the  evaluative  factor  with  the 
term  "attitude"  because  nearly  all  of  the  adjective  pairs 
imply  negative  and  positive  characteristics  (Osgood,  Suci, 
and  Tannenbaum,  1957:189;  Brinton,  1961:289;  Shaw  and 
Wright,  1967:30;  Nunnally,  1967:537).  Osgood  and  his 
associates  reported  the  results  of  two  studies  designed  to 
evaluate  the  use  of  the  semantic  differential  as  a measure 
of  attitude.  Both  studies  employed  multiple  measurements: 
the  semantic  differential,  Thurstone  scales,  and  a Guttman 
scale.  Correlations  with  the  Thurstone  scales  ranged  from 
.74  to  .84.  The  correlation  of  Guttman  scale  scores  with  a 
three  item  semantic  differential  scale  score  was  .79.  The 
conclusion  was  that  whatever  the  Thurstone  and  Guttman 
scales  were  measuring,  the  evaluative  factor  of  the  semantic 
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differential  was  also  measuring.  The  authors  concluded 
that : 


The  findings  of  both  of  these  studies  support  the 
notion  that  the  evaluative  factor  of  the  semantic 
differential  attitude.  It  is,  moreover,  a method  of 
attitude  assessment  that  is  relatively  easy  to 
administer  and  easy  to  score  (Osgood,  Suci,  and 
Tannenbaum,  1957:194). 

The  method  of  constructing  a semantic  differential  for 
attitude  measurement  is  as  follows : 

1.  Name  the  objects  or  concepts  that  are  to  be 
measured. 

2.  Assemble  adjective  pairs  which  are  indicative  of 
evaluative  characteristics  for  each  object  or  concept  and 
form  bipolar  scales. 

3.  Assign  numbers  to  the  scale  intervals  to  indicate 
direction  and  intensity.  Scores  for  each  object  or  concept 
are  obtained  by  summing  the  associated  scale  scores. 

The  versatility  of  the  semantic  differential  technique 
was  documented  by  the  extensive  number  of  researchers  who 
applied  it  to  a wide  range  of  research  problems.  A bibliog- 
raphy of  the  literature  relevant  to  the  semantic  differen- 
tial technique  revealed  that  over  one  thousand  books  and 
articles  were  published  (Bobren,  Hill,  Snider,  and  Osgood, 
1968) . Haried  (1969)  reported  that  over  six  hundred  diverse 
studies  were  completed  using  the  semantic  differential 
technique. 
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The  technique  seemed  ideally  suited  for  multi- 
dimensional constructs  because  different  factors  could  be 
considered  as  concepts  to  provide  independent  factor  scores. 
Furthermore,  the  complexity  of  certain  factors  could  be 
accommodated  through  judicious  selection  of  the  adjective 
pairs.  The  bipolarity  of  the  technique  matched  the  bipolar 
nature  of  the  user  satisfaction  construct.  There  were  no 
immediately  obvious  disadvantages  to  adapting  the  semantic 
differential  for  measuring  user  satisfaction;  however,  a 
more  in-depth  review  was  conducted  prior  to  the  methodology 
selection. 


EVALUATION  OF  THE  SEMANTIC  DIFFERENTIAL  TECHNIQUE 
It  was  previously  stated  that  the  semantic  differential 
was  developed  to  measure  the  "meaning"  of  concepts.  The 
technique  was  based  on  the  role  of  adjectives  in  the 
communication  process.  In  order  to  understand  "what"  the 
semantic  differential  measures,  the  different  facets  of 
"meaning"  were  distinguished.  Oliver  stated: 

The  most  common  communication,  linguistic,  has  two 
aspects  to  its  meaning  - connotative  and  denotative. 
Denotative,  or  referential,  word  meaning  identifies  the 
sensorially  relevant  characteristics  (such  as  dimension 
and  weight;  of  that  being  identified.  Since  these 
characteristics  are  descriptive,  they  generally  lack 
emotional  overtones. 

On  the  other  hand,  and  behaviorally  more  important, 
words  also  possess  connotative  meaning.  This  is  the 
affective  aspect  of  linguistic  communication.  It  is  the 
aspect  of  a communication  which  produces  emotional 
reactions  (interpretations).  An  opinion,  belief,  or 
attitude  is  molded  in  terms  of  each  individual's 
connotative  meanings  (Oliver,  1973:549). 
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Nunnally  (1967)  identified  association  as  a third  facet 
of  meaning  that  overlaps  with  denotation  and  connotation. 
"Associations"  were  those  objects  that  were  brought  to  mind 
when  an  individual  saw  or  heard  about  a particular  object. 

Of  these  three  aspects  of  meaning,  only  connotation  was 
important  for  attitude  measurement. 

Osgood,  Suci,  and  Tannenbaum  (1957)  were  very  explicit 
that  the  semantic  differential  was  developed  to  measure 
connotative  feelings  about  concepts.  Crockett  and  Nidorf 
(1967:211)  concluded  that  "the  instrument  [the  semantic 
differential)  is  wedded  to  a detailed  and  persuasive  theory 
of  connotative  meaning."  Therefore,  the  semantic  differ- 
ential was  intended  to  extract  responses  that  conveyed 
information  about  an  individual's  feelings  toward  a 
measurement  object  on  the  basis  of  what  that  object  meant 
connotatively  to  that  individual.  The  concensus  of  most 
reviewers  was  summed  up  by  Nunnally: 

The  semantic  differential  mainly  measures 
connotative  aspects  of  meaning,  particularly  the 
evaluative  connotations  of  objects.  For  that  purpose, 
it  probably  is  the  most  valid  measure  of  connotative 
meaning  available.  Because  of  the  nature  of  the 
instrument,  it  cannot  measure  nonconnotative  asso- 
ciations... The  evaluative  factor  of  the  semantic 
differential  is  almost  purely  connotative  in  character 
rather  than  denotative  or  associative.  This  is  why  it 
was  said  that  the  evaluative  factor  should  provide  a 
good  measure  of  attitudes  (Nunnally,  1967:541). 

It  was  concluded  that  the  semantic  differential  did 
measure  the  appropriate  aspect  of  meaning  for  the  user 
satisfaction  construct.  Although  the  semantic  space 
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associated  with  the  semantic  differential  was  shown  to  be 
multi-dimensional,  only  the  Evaluation  dimension  was  used  to 
measure  attitudes  (Osgood,  1962;  Shaw  and  Wright,  1967; 
Heise,  1970). 


Metric  Properties  of  the  Semantic  Differential 

Traditional  use  of  the  semantic  differential  was  based 
on  two  assumptions  about  the  metric  properties  of  the 
individual  bipolar  scales.  These  assumptions  were  that  the 
intervals  within  the  scale  were  equal  and  that  the  zero 
point  occurred  at  the  same  place  on  each  scale,  namely  the 
midpoint.  Messick  (1957)  utilized  the  method  of  successive 
intervals  to  evaluate  these  assumptions. 

The  seven  interval  scale  used  by  Osgood,  Suci,  and 
Tannenbaum  (1957)  was  assigned  the  integers  3,  2,  1,  0,  -1, 
-2,  and  -3  as  successive  category  midpoints;  therefore,  the 
assumed  boundary  values  for  the  response  categories  were 
2.5,  1.5,  .5,  -.5,  -1.5,  -2.5.  Since  the  method  of  succes- 
sive intervals  provided  scale  values  for  the  boundaries  of 
the  response  categories,  Messick  (1957)  was  able  to  compare 
the  actual  with  the  assumed  boundary  values  for  the  nine 
most  frequently  used  bipolar  scales.  The  results  indicated 
that,  while  the  intervals  did  not  appear  to  be  strictly 
equal  within  scales,  the  intervals  did  have  a consistent 
size  between  scales.  Also,  the  zero  point  occurred  at  the 
same  approximate  position  on  each  scale,  a position  slightly 
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on  the  negative  side  of  the  assumed  zero  point.  Further 
analysis  indicated  that  the  interval  variations  were  within 
the  limits  that  could  have  occurred  due  to  random  error 
fluctuations . 

Messick  (1957)  then  examined  the  effect  of  assuming 

equality  rather  than  using  the  boundary  values  established 

by  the  successive  intervals  procedures.  The  correlations 

between  the  two  scales  for  the  nine  adjective  pairs  ranged 

from  .984  to  .998.  The  results  led  Messick  to  conclude: 

. . . that  little  distortion  would  be  introduced  by  using 
successive  intervals  as  category  mid-points  for  these 
nine  scales.  Considering  this  and  other  indications  of 
the  present  study,  i.e.,  an  approximate  equality  of 
corresponding  interval  lengths  from  scale  to  scale  and  a 
similar  placement  of  origins  across  scales,  it  seems 
reasonable  to  conclude  that  the  scaling  properties 
implied  by  the  Semantic  Differential  procedures  have 
some  basis  other  than  mere  assumption  (Messick, 

1957:206) . 

Messick' s conclusions  were  further  supported  by  Cliff 
(1959) . The  seven  response  categories  for  each  bipolar 
scale  were  labeled  by  the  adverb  qualifiers  extremely, 
quite,  slightly,  neither  or  equally,  slightly,  quite,  and 
extremely  which  combined  multiplicatively  with  the  polar 
terms  of  the  scale.  Cliff  found  that  this  particular  set  of 
adverb  qualifiers  combined  with  adjective  sets  to  form 
combinations  of  nearly  equal  step  functions  of  intensity  of 
meaning.  Thus,  the  response  categories  varied  in  intensity 
of  feeling  in  near  equal  intervals. 
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McCroskey,  Prichard,  and  Arnold  (1967)  examined  the 
assumption  of  the  origin  placement  at  the  scale  midpoint. 
The  findings  refuted  the  assumption  for  individual  bipolar 
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scales,  but  supported  it  for  summated  scale  scores.  The 
writers  stated: 

It  appears  that  although  the  midpoint  of  a single 
bipolar  scale  may  not  represent  the  neutral  point  in 
attitude  intensity,  the  midpoint  of  the  range  of 
possible  summated  scores  across  six  bipolar  scales  does 
represent  the  neutral  point  in  attitude  intensity 
(McCroskey,  Prichard,  and  Arnold,  1967:644). 

The  question  of  whether  the  semantic  space  was  bipolar 
and  whether  the  scales  of  the  semantic  differential  accu- 
rately reflected  this  polarity  was  the  subject  of  a study  by 
Green  and  Goldfried  (1965).  The  results,  based  on  unipolar 
scale  ratings,  revealed  that  the  scale-concept  interaction 
had  a great  effect  not  only  on  the  bipolarity  but  also  on 
the  symmetry  of  the  scales  anchored  by  opposing  adjectives. 
Bipolar  factors  were  found  for  the  Evaluation  dimension  of 
the  semantic  space  but  not  for  the  dimensions  of  Potency  and 
Activity. 

These  results  were  refuted  in  part  by  a similar  study 
performed  earlier  by  Taylor  and  Kumata  (cited  by  Osgood, 
Suci,  and  Tannenbaum,  1957:153).  Heise  (1969)  observed  that 
the  results  obtained  by  Green  and  Goldfried  would  have  indi- 
cated bipolarity  if  the  unipolar  rating  had  been  averaged. 
Overall,  it  was  felt  that  the  assumption  of  bipolarity, 
particularly  for  the  Evaluation  dimension,  was  justified. 


The  studies  that  were  reviewed  supported  the  following 
assumptions  about  the  metric  properties  of  the  semantic 
differential.  When  the  individual  dimensions  of  the  seman- 
tric  space  were  examined  independently,  the  Evaluation 
dimension  received  the  strongest  evidence  for  these 
assumptions : 

1.  The  scale  intervals  were  equal. 

2.  The  point  of  neutral  attitude  intensity  was  located 
at  the  midpoint  of  the  range  of  the  summated  scale  values 
for  factor  scores. 

3.  The  adjective  pairs  did  anchor  a bipolar  continuum. 

Reliability  of  the  Semantic  Differential 

Traditional  discussions  of  reliability  as  a property  of 
measurement  instruments  centered  on  the  reproduction  of 
scores  (Cronbach,  1951) . The  methods  for  testing  this  prop- 
erty typically  included  test-retest  procedures,  split-half 
methods,  and  parallel  forms  of  the  same  type  of  instrument 
(Summey,  1974) . Each  of  these  methods  was  predicated  on  the 
development  of  reliability  coefficients  from  correlations  of 
the  results  of  at  least  two  different  test  scores.  More 
recently,  these  types  of  reliability  coefficients  considered 
as  indicators  of  stability  rather  than  reliability  in  the 
sense  of  the  accuracy  or  precision  of  the  instrument 
(Kerlinger,  1973). 
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The  conception  of  reliability  as  the  accuracy  of  the 
measurement  instrument  was  a fundamental  departure  from  the 
reproduction  of  scores  viewpoint.  This  approach  to  the 
determination  of  reliability  was  based  on  a theory  of  meas- 
urement error  (Thorndike,  1967;  Nunnally,  1967;  Kerlinger, 
1973)  . Reliability  was  defined  as  the  relative  absence  of 
errors  of  measurement  in  the  measurement  instrument. 

Kerlinger  (1973)  presented  the  rationale  for  a defini- 
tion of  reliability  in  terms  of  errors  of  measurement.  For 
any  set  of  scores  obtained  by  a measurement  instrument,  the 
total  variance  (V)  was  considered  to  be  composed  of  the  sum 
of  the  true  variance  (Vt)  and  the  error  variance  (V  ) , i.e., 
V = Vt  + Ve.  Therefore,  the  reliability  of  a measurement 
instrument  was  indicated  by  the  proportion  of  true  variance 
to  the  total  variance.  The  equation  for  this  calculation 
was  given  by  Kerlinger  (1973:446)  as 

rtt  = 1 - (2.3) 

ir 

where 

rfct  = the  reliability  coefficient 

V = total  variance  computed  from  the  obtained  set  of 
scores 

V0  = error  or  residual  variance 

Kerlinger' s equation  was  computationally  equivalent  to 


one  proposed  earlier  by  Hoyt  (1941) . Hoyt  utilized  an 
analysis  of  variance  approach  and  defined  reliability  by  the 
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expression: 

Error  Variance  (2.4) 

Reliability  = 1 - Variance  among  individuals 

Thorndike  observed: 

The  analysis  of  variance  approach  appears  useful  for 
obtaining  reliability  estimates  from  items  or  trials 
which  are  scored  with  a range  of  scores,  and  not  merely 
"passed"  or  "failed"  (Thorndike,  1967:238). 

The  semantic  differential  was  subjected  to  studies 
directed  at  both  the  stability  and  accuracy  aspects  of 
reliability.  DiVesta  and  Dick  (1966)  studied  the  test- 
retest  ratings  of  semantic  differential  scales  by  grade 
school  children.  A four  week  time  lapse  was  used  between 
rating  periods.  The  correlations  between  the  two  sets  of 
scores  were  somewhat  higher  for  the  higher  grades  and  also 
for  the  Evaluation  dimension  overall.  The  factor  score 
correlations  for  the  Evaluation  dimension  for  grades  two 
through  seven  were  .39,  .49,  .75,  .86,  .79,  and  .78, 
respectively.  DiVesta  and  Dick  concluded: 

The  findings  in  the  present  study  clearly  suggest 
that  over  brief  periods,  the  semantic  differential  is  a 
reliable  instrument  when  used  with  children.  The 
coefficients  of  stability  are  comparable  to  those 
ordinarily  obtained  by  other  techniques  requiring 
judgments  by  individuals  (DiVesta  and  Dick,  1966:613). 

Two  other  conclusions  from  the  study  merited  attention: 

The  reliability  of  the  data  generated  by  the 
semantic  differential  is  higher  when  ratings  from  two 
scales  are  combined  to  obtain  factor  scores  than  when 
only  single  scales  are  used...  For  usual  research 
practice,  where  the  concern  is  with  only  one  to  three 
dimensions,  it  seems  likely  that  the  investigator  will 
choose  to  combine  the  scores  from  three  to  five  scales 
in  order  to  realize  higher  stability...  Reliability  for 
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concepts  based  on  group  means  is  higher  than  that  for 
individuals  even  when  groups  are  composed  of  only  three 
to  five  subjects  (DiVesta  and  Dick,  1966:613). 


A test-retest  study  by  Norman  (1959)  examined  the 
shifts  in  ratings  compared  to  expected  shifts  if  the  ratings 
were  purely  random.  The  results  showed  that  40  percent  of 
the  scale  ratings  did  not  shift,  35  percent  of  the  ratings 
shifted  by  one  scale  unit,  and  25  percent  of  the  ratings 
shifted  two  or  more  scale  units.  Some  concepts  had  more 
stable  ratings  than  others.  This  difference  was  apparently 
related  to  the  extremeness  of  the  concept.  Peabody  (1962) 
also  found  that  more  neutral  concepts  were  rated  with  less 
stability. 

It  was  concluded  that  the  semantic  differential  did 
achieve  an  acceptable  degree  of  stability  for  both  indi- 
vidual and  group  mean  scores,  although  the  group  means  were 
significantly  more  stable.  The  Evaluation  dimension  scales 
demonstrated  more  stability  in  test-retest  correlations  than 
the  other  dimensions.  Notwithstanding  these  results, 

Osgood,  Suci,  and  Tannenbaum  argued  against  the  use  of 
correlation  type  measures  as  inappropriate  for  the  semantic 
differential : 

. . . the  correlation  coefficient  does  not  take  into 
account  the  absolute  differences  between  the  means  of 
the  two  tests;  perfect  reliability,  "r"  of  1.00  can 
occur  when  an  absolute  difference  of  several  units 
exists  between  test  and  retest  measurements  such  that 


not  a single  score  is  reproduced  (Osgood,  Suci,  and 
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Summey  (1974)  used  the  Kerlinger  (1973)  equation  to 
evaluate  the  semantic  differential  reliability  as  a function 
of  the  errors  of  measurement  in  a study  of  business  stud- 
ents' attitudes.  The  measurement  instrument  achieved  an 
value  of  .814  which  meant  that  81.4  percent  of  the  total 
variance  of  the  measured  variables  was  accounted  for  by  the 
measurement  technique  (Summey,  1974:64).  This  level  of 
reliability  was  comparable  to  reliability  coefficients 
developed  in  similar  research  (Heise,  1969). 

The  use  of  the  Kerlinger  (1973)  equation  was  considered 
to  be  highly  dependent  on  the  particular  instrument  being 
evaluated.  While  the  results  of  the  stability  studies  may 
be  generalized  because  the  research  was  directed  at  the 
semantic  differential  methodology  itself,  reliability  coef- 
ficients based  on  a particular  measurement  instrument  were 
not  generalizable.  Therefore,  coefficients  to  estimate  the 
accuracy  aspect  of  the  reliability  property  must  be  calcu- 
lated for  any  measurement  instrument  being  considered.  The 
stability  aspect  of  the  semantic  differential,  however,  was 
supported  by  numerous  studies.  Stability  increased  when 
multiple  scales  were  used  for  factor  scores,  and  group  means 
were  more  stable  than  individual  ratings.  The  Evaluation 
dimension  was  repeatedly  more  stable  across  both  concepts 
and  individuals  than  the  other  semantic  space  dimensions. 
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Validity  of  the  Semantic  Differential  Technique 

A measurement  instrument  is  said  to  be  valid  when  it 
measures  what  it  is  supposed  to  measure  (Osgood,  Suci,  and 
Tannenbaum,  1957:140).  Validity  has  traditionally  been  a 
difficult  property  to  document  for  measurement  instruments 
developed  for  pyschometric  purposes.  Nunnally  (1967:75) 
acknowledged  that  "validity  is  a matter  of  degree  rather 
than  an  all-or-none  property,  and  validation  is  an  unending 
process . " 

Three  categories  of  measurement  validity  have  evolved, 
and  these  are  labeled  (1)  predictive  validity,  (2)  content 
validity,  and  (3)  construct  validity  (Ghiselli,  1964; 
Nunnally,  1967) . Predictive  validity  is  signified  by  the 
correlation  between  the  measurement  scores  and  some  inde- 
pendent criterion  that  is  assumed  to  represent  the  "true" 
state  of  the  attribute  being  measured.  The  absence  of  an 
independent  criterion  has  frequently  limited  the  use  of 
predictive  validity  measures  for  psychometric  measurement 
instruments  designed  to  evaluate  "how  much"  of  an  attribute 
is  present  in  a subject  (Nunnally,  1967). 

Content  validity  is  determined  from  the  content  and 
operations  of  a measurement  instrument  (Ghiselli,  1964) . 
Validity  in  this  sense  implies  that  the  content  of  the 
construct  has  been  adequately  sampled  from  the  construct 
domain  and  that  the  operations  to  measure  this  content  are 


"reasonable".  Determination  of  how  well  these  two  require- 
ments are  satisfied  is  often  based  on  logical  and  subjective 
evidence . 

One  approach  to  minimize  the  effects  of  subjective 
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judgments  has  been  the  utilization  of  the  homogeneity  of  the 
items  of  the  instrument  as  an  indication  of  validity 
(Robinson,  Rush,  and  Head,  1974).  Homogeneity,  also  re- 
ferred to  as  internal  consistency,  is  based  on  the  argument 
that  component  parts  of  an  instrument  that  purport  to 
measure  the  same  thing  should  be  positively  correlated. 
Therefore,  high  intercorrelations  are  indicative  of  content 
validity. 

Construct  validity  is  based  on  the  explanation  of 
measurement  scores  within  some  network  of  theoretical 
variables.  This  type  of  validity  is  involved  primarily  when 
the  scores  are  to  be  interpreted  in  terms  of  an  attribute 
that  has  not  been  operationally  defined  (Cronbach  and  Meehl , 
1955) . Construct  validity  is  determined  by  the  behavior  of 
scores  in  relationship  to  other  variables  theoretically 
contained  within  the  construct  network.  Validity  is  a func- 
tion of  the  relationships  following  a postulated  pattern. 

Although  three  types  of  validity  are  distinguished,  in 


practice  the  definitions  overlap  and  different  measurement 
problems  pose  unique  characteristics  in  terms  of  validity 
documentation.  As  with  reliability,  validity  is  viewed  as  a 
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property  determined  by  logical  as  well  as  statistical  argu- 
ments. Measurement  instruments  based  on  the  semantic 
differential  technique  have  been  validated  in  a number  of 
ways . 

Smith  (1963)  used  a criterion-oriented  method  to 
validate  a semantic  differential  instrument  for  rating 
public  speaker  effectiveness.  A "known  group"  approach 
utilized  instructor  ratings  of  students  enrolled  in  speech 
courses  to  differentiate  the  students  in  terms  of  public 
speaking  ability.  These  students  were  then  rated  using  a 
semantic  differential  instrument.  The  high  correlations 
between  the  two  ratings  were  considered  as  evidence  for 
validity. 

Grigg  (1959)  utilized  a construct  validity  approach  to 
validate  the  semantic  differential  technique.  The  study 
involved  the  expected  distances  between  subjects'  concepts 
of  "self",  "ideal  self",  and  "neurotic"  both  before  and 
after  experimental  manipulation.  The  expected  relationships 
did  occur  (the  construct  network  behavior) , and  Grigg  con- 
cluded that  the  evidence  supported  the  validity  claim  of  the 
semantic  differential. 

Summey  (1974)  employed  a content  validity  technique  to 
evaluate  a semantic  differential  for  attitude  measurement  of 
business  students.  The  validity  claim  was  based  on  careful 
articulation  of  the  content  development  and  actual  construc- 
tion of  the  instrument.  Factor  analysis  was  then  used  to 


substantiate  the  selection  of  scales  to  measure  the  hypothe- 
sized factors. 


These  three  examples  demonstrated  the  use  of  different 
approaches  to  validate  semantic  differential  instruments. 
They  were  representative  of  a number  of  studies  claiming  the 
validity  of  the  semantic  differential  technique.  Those 
specifically  concerned  with  attitude  measurement  included: 

1.  Attitude  formation  (Barclay  and  Thumin,  1963). 

2.  Attitudes  toward  organizations  (Rodefeld,  1967). 

3.  Attitudes  toward  jobs  and  occupations  (Triandis, 
1959;  Gusfield  and  Swartz,  1963). 

4.  Attitudes  toward  minorities  (Prothro  and  Keehn, 
1957;  Williams,  1966). 

The  evidence  was  compelling  that  validity  could  be 
achieved  in  attitude  measurement  instruments  based  on  the 
semantic  differential  technique.  This  potential,  however, 
was  not  automatically  achieved.  Block  (1958)  reported 
results  unfavorable  to  the  use  of  the  semantic  differential 
in  measuring  one's  "ideal  self".  Nichols  and  Shaw  (1964) 
found  that  validity,  as  determined  by  correlations  with 
Thurstone  scales,  was  reduced  as  the  saliency  of  the  con- 
cepts being  measured  was  increased.  The  contradictory 
evidence  confirmed  the  requirement  that  each  measurement 
instrument  be  evaluated  for  validity  in  accordance  with  the 
measurement  purpose. 


Sources  of  Errors  in  Semantic  Differential  Ratings 

In  addition  to  the  errors  of  measurement  reflected  in 
the  reliability  of  an  instrument,  other  factors  were 
identified  as  potential  sources  of  error  in  the  semantic 
differential  ratings.  These  were: 

1.  Response  bias 

2.  Anchoring  effects 

3.  Scale-concept  interaction 

Response  bias  was  the  alteration  of  a response  to  a 
measurement  item  in  such  a way  that  it  indicates  something 
other  than  what  the  item  was  intended  to  measure  (Guilford, 
1967) . The  biases  of  interest  were  those  determined  by 
mental  sets,  termed  response  sets,  which  existed  in  the 
subjects  completing  the  measurement  instrument.  Heise 
(1969)  categorized  the  response  sets  most  affecting  semantic 
differential  ratings  as  (1)  social  desirability  and  (2) 
scale-checking  style. 

Nichols  and  Shaw  (1964)  found  that  the  saliency  of 
concepts  being  measured  was  inversely  related  to  the 
reliability  of  the  semantic  differential  scores.  This 
result  was  attributed  to  subjects'  sensitivity  to  social 
repercussions  of  their  ratings  and  to  the  fact  that  the 
semantic  differential  was  an  "obvious"  measure  of  attitude, 
i.e.,  not  disguised.  This  result  was  supported  by  the 
findings  of  a study  by  Ford  and  Meisels  (1965)  who  found 
high  correlations  between  independent  measures  of  the  social 
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desirability  of  semantic  differential  scales  and  the  factor 


analysis  loadings  of  the  scales  on  the  Evaluation  dimension. 

The  conclusion  suggested  by  Keise  (1969)  was  that  the 
semantic  differential  may  not  be  especially  well  suited  for 
the  measurement  of  socially  sensitive  topics.  It  was  not 
felt  that  the  user  satisfaction  construct  was  one  of  such 
"social  sensitivity"  that  this  type  of  response  set  would 
greatly  affect  the  scores  obtained  from  semantic  differ- 
ential ratings. 

The  response  set  identified  as  scale- checking  style  was 
found  by  Peabody  (1962)  to  be  important  for  semantic  differ- 
ential usage.  Specifically,  he  found  that  some  subjects 
appeared  to  use  the  end  positions  of  the  scale  responses 
more  often  and  to  avoid  the  use  of  the  intermediate  posi- 
tions. After  a comparison  of  abnormal  and  normal  subject 
group  semantic  differential  ratings,  Arthur  (1966)  confirmed 
that  subjects  in  the  abnormal  groups  had  a significantly 
higher  proportion  of  extreme  responses  than  normal  subjects. 
Heise  reviewed  14  studies  which  examined  the  scale-checking 
style  of  subject  groups  differentiated  by  age,  IQ,  F scores 
(a  measure  of  authoritarianism),  sex,  neurosis,  and 
psychosis,  and  he  concluded: 

The  available  research  documents  the  existence  of 
basic  differences  between  people  in  the  extent  to  which 
extremes  are  checked.  Therefore,  exaggeration  bias 
should  be  a considered  variable  in  experiments, 
controlled  either  by  random  assignment  of  subjects  into 
experimental  and  control  groups  or  by  attempting  to 
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measure  the  effect  in  order  to  control  for  it 
statistically  (Heise,  1969:409). 

This  type  of  response  set  could  distort  the  inter- 
pretation of  semantic  differential  scores;  however,  it  was 
an  unresolved  issue  whether  the  bias  actually  represented  a 
variation  from  the  "true"  score  or  was  a measure  of  a 
subject's  propensity  to  exaggerate  which  perhaps  should  be 
considered  as  a confounded  part  of  a subject's  attitude 
(Nunnally,  1967) . The  nature  of  this  type  of  response  set 
precluded  any  attempt  to  "extract"  this  bias  from  a semantic 
differential.  It  was  decided  to  follow  the  advice  of 
Guilford  (1967)  and  attempt  to  control  the  effects  of 
response  sets  through  the  use  of  good  measurement  instrument 
formats  and  clear,  appropriate  instructions  suitable  for  the 
subject  population. 

Another  potential  source  of  error  was  the  effect  of 
anchoring,  sometimes  referred  to  as  context,  on  the  semantic 
differential  ratings  (Torgerson,  1958).  Anchoring  was  the 
tendency  of  a subject,  while  rating  a list  of  stimuli,  to 
assimilate  carry-over  effects  from  one  or  more  stimuli  to 
the  next.  In  other  words,  the  order  of  the  stimuli  could 
affect  the  ratings  assigned  to  an  individual  stimulus  within 
the  set. 

Somer  (1965)  examined  this  effect  on  semantic  differ- 
ential ratings  by  preceding  test  items  with  positive  and 
negative  contexts.  One  example  was  the  test  item,  winter, 


which  was  preceded  by  either  hate,  steal,  and  devil 
(negative  context)  or  by  faith,  peace,  and  sunlight 
(positive  context).  The  test  results  indicated  that  the 
semantic  differential  ratings  were  very  resistant  to 
anchoring  effects.  These  findings  agreed  with  an  earlier 
study  conducted  by  Aiken  (cited  by  Osgood,  Suci,  and 
Tannenbaum,  1957) . 

There  was  one  other  potential  source  of  error.  After 
ten  years  of  concentrated  research  devoted  to  the  methodo- 
logy, application,  and  theory  of  the  semantic  differential, 
Osgood  (1962:24)  stated  that  "the  preceding  evidence  as  a 
whole  forces  us  to  the  conclusion  that  there  is  significant 
interaction  between  concepts  and  scales  in  the  process  of 
semantic  judgement." 

It  was  felt  that  the  scale-concept  interaction  phenome 
non  was  the  most  important  potential  source  of  error  in  a 
semantic  differential  designed  to  measure  user  satisfaction 
As  Nunnally  (1967:539)  pointed  out,  "whereas  'rugged'  is 
positively  evaluative  when  applied  to  men,  it  is  not  posi- 
tively evaluative  when  applied  to  women."  Brown's  (1958) 
classic  question,  "Is  a boulder  sweet  or  sour?",  was  a 
succinct  expression  of  the  crux  of  the  scale-concept 
interaction  problem. 

The  selection  of  scales  (bipolar  adjective  pairs)  for 
measuring  a concept  should  satisfy  three  criteria:  (1) 
factorial  composition,  (2)  relevance,  and  (3)  semantic 
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stability  (Osgood,  Suci,  and  Tannenbaum,  1957;  Summey, 

1974) . The  factorial  composition  criterion  requires  that 
the  scales  be  highly  loaded  on  a particular  factor  (semantic 
space  dimension)  being  measured  and  minimally  loaded  on  all 
other  factors  which  may  be  elements  of  the  concept.  These 
loadings  may  be  determined  by  factor  analysis.  The 
relevance  criterion  requires  that  the  adjective  pairs  be 
appropriate  descriptors  for  the  concept  being  measured.  The 
criterion  of  semantic  stability  is  indicated  by  the  lack  of 
confusion  about  how  a scale  applies  to  a particular  concept. 
Careful  adherence  to  these  three  criteria  during  the  scale 
selection  process  should  minimize  any  errors  caused  by 
scale-concept  interactions.  As  stated  by  Heise: 

The  existence  or  possible  existence  of  concept-scale 
interaction,  whether  it  is  a function  of  relevance  or 
stimulus  environment,  means  that  a semantic  differential 
ideally  should  be  validated  and  adjusted  for  every  new 
stimulus  class  with  which  it  is  used. . . More  precise 
measurements  will  be  attained  only  by  tailoring  instru- 
ments to  each  content  domain  so  as  to  control  for  true 
concept-scale  interactions  (Heise,  1969:418). 

Advantages  of  the  Semantic  Differential 

In  addition  to  the  evidence  for  the  semantic  differen- 
tial methodology  as  reliable,  valid,  and  robust  against 
rating  errors,  other  advantages  for  the  use  of  the  semantic 
differential  to  measure  user  satisfaction  were  noted.  These 
advantages  were: 

1.  No  required  subject  verbalization 

2.  Natural  anchor  points 
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3.  Direct  interpretation  of  scores 

4.  Economical  instrument  construction  and  modification 

5.  Economical  subject  test  time 

6.  Adaptability  for  weighting  functions 

7.  Objectivity 

Kaufman  stated,  "Major  assets  of  the  semantic  differ- 
ential include  the  fact  that  it  requires  no  verbalization  on 
the  part  of  the  respondents  and  that  it  measures  emotional 
reactions  rather  than  rational  or  well-reasoned  ones"  (cited 
by  Klapper,  1959:437).  The  lack  of  required  subject 
verbalization  was  perceived  as  very  important  in  view  of  the 
findings  of  Fingeld  (1966)  which  revealed  that  the  ability 
of  persons  to  select  words  to  convey  intended  meanings 
showed  wide  variance.  Moreover,  no  significant  relationship 
was  found  between  the  ability  to  select  words  and  a 
subject's  general  intelligence,  verbal-reasoning  ability, 
vocabulary,  sex,  or  advanced  training  in  speech.  The 
semantic  differential  format  which  precluded  subject 
articulation  of  meaning,  therefore,  required  careful 
selection  of  scales  for  each  concept  being  measured. 

The  measurement  model  specified  in  Equation  2.2  was 


n 


This  model  wan  predicated  upon  the  establishment  of  attitude 
continuums  for  each  facet  of  the  satisfaction  construct.  A 
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meaningful  continuum  had  to  be  defined  with  anchor  points 
that  were  acceptable  to  a wide  variety  of  users  which  com- 
prised the  population  of  interest.  The  semantic  differen- 
tial established  natural  anchor  points  which,  when  combined 
with  the  adverb  "extremely",  were  acceptable  to  a wide 
variety  of  potential  subjects  (Cliff,  1959). 

The  semantic  differential  scores  were  subject  to  direct 
interpretation.  The  assignment  of  integer  values  to  the 
response  categories  established  maximum  and  minimum  values 
for  the  continuum.  Utilization  of  average  scores  for  each 
facet  of  the  construct,  scale  sums  divided  by  the  number  of 
scales,  permitted  direct  interpretation  of  a subject's 
relative  position  on  a particular  continuum.  In  contrast, 
Likert- type  scale  scores  required  normalization  within  a 
particular  subject  group  for  meaningful  interpretation. 

The  construction  of  a semantic  differential  was  con- 
sidered more  economical  than  either  Thur stone,  Likert,  or 
Guttman  scale  instruments.  This  advantage  primarily  stemmed 
from  not  requiring  a large  number  of  judges  to  sort  and 
assign  values  to  many  statements  reflecting  the  content 
domain.  The  semantic  differential  format  also  allowed  rapid 
modification,  either  reduction  or  enhancement,  to  reflect 
unique  characteristics  of  the  subject  group  or  measurement 
puupose.  Although  any  such  rapid  modifications  may  lack 
sophisticated  confirmation  of  validity  and  reliability 
properties,  adherence  to  the  scale  selection  criteria  could 
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allow  measurement  studies  to  be  undertaken  with  a reasonable 
degree  of  confidence. 

Kiesler,  Collins,  and  Miller  (1969)  pointed  out  that 
fatigue  factors  caused  by  excessive  instrument  length, 
particularly  in  uncompensated  subjects,  could  greatly 
increase  measurement  errors.  The  semantic  differential 
allowed  data  to  be  collected  rapidly.  Osgood,  Suci,  and 
Tannenbaum  (1957)  reported  that  subjects  could  easily 
complete  400  judgments  per  hour.  Heise  (1970)  challenged 
this  rate  as  very  generous  and  suggested  that  even  the 
slowest  of  subjects  could  make  judgments  at  a more  rapid 
rate.  This  allowed  a very  large  amount  of  information  to  be 
collected  per  unit  time  from  individual  subjects. 

The  semantic  differential  format  was  also  amenable  to 
individual  factor  weighting.  The  addition  of  a single  scale 
for  importance  allowed  the  subject  to  indicate  the  saliency 
of  each  particular  concept  (Mobley  and  Locke,  1970). 
Weighting  functions  could  then  be  established  by  assignment 
of  weights  to  each  response  category  for  the  importance 
scale. 

The  semantic  differential  required  no  subjective  judg- 
ment on  the  part  of  the  investigator  administering  the 
instrument.  The  subject  responses  were  directly  quanti- 
fiable. Thus,  the  objectivity  of  the  semantic  differential 
yielded  verifiable,  reproducible  data  from  all  subjects 
completing  the  ratings.  This  advantage  allowed  cross-study 
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comparisons  which  facilitated  the  research  process. 

SUMMARY 

The  user  satisfaction  construct  was  established  as  a 
multi- dimensional  attitude.  A measurement  model  that 
reflected  the  multiple  facets  of  the  construct  was  proposed. 
The  model 

Si  = I W±j  Rij 
j=l 

was  additive  with  each  factor  score  representing  a subject's 
position  on  an  underlying  continuum.  Each  factor  was  to  be 
weighted  to  express  the  saliency  of  a particular  factor  to 
an  individual  sub j ect . 

A review  of  traditional  measurement  scales  revealed  the 
advantages  of  achieving  at  least  an  interval  scale  with  the 
measurement  process.  The  scaling  methodologies  most  often 
utilized  in  the  measurement  of  attitudes  were  examined,  and 
the  semantic  differential  technique  was  selected  as  the  most 
appropriate  for  measuring  user  satisfaction. 

This  selection  was  based  on  the  pervasive  theoretical 
and  empirical  evidence  that  the  semantic  differential, 
properly  constructed  for  the  content  domain,  resulted  in  a 
reliable,  valid,  and  robust  instrument  for  the  measurement 
of  attitudes.  The  metric  properties  of  the  semantic  dif- 
ferential scales  were  demonstrated  as: 
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1.  The  scale  was  an  interval  scale. 

2.  The  zero  point  of  attitude  intensity  was  located  at 
the  midpoint  of  the  range  of  summated  scales . 

3.  The  scale  was  a bipolar  continuum. 

Other  advantages  were  identified  that  also  contributed 
to  the  selection.  Although  the  semantic  differential  meth- 
odology was  considered  relatively  straightforward,  the 
importance  of  scale  selection  of  bipolar  adjective  pairs  was 
noted.  Adjective  pairs  utilized  in  a semantic  differential 
should  satisfy  the  criteria  of: 

1.  Factorial  composition 

2.  Relevance 

3.  Semantic  stability 

The  analysis  of  the  underlying  assumptions  established 
the  foundations  for  the  conduct  of  this  research  study.  The 
results  of  this  chapter  gave  insight  into  the  measurement 
problem  and  described  the  measurement  model.  The  results 
also  provided  the  fundamental  basis  to  answer  the  question 
of  "how"  to  measure  the  user  satisfaction  construct.  The 
next  chapter  defines  the  content  domain  by  distinguishing 
the  different  factors  which  comprise  the  user  satisfaction 
construct. 


Chapter  3 


DOMAIN  OF  THE  USER  SATISFACTION  CONSTRUCT 


This  definition  of  the  user  satisfaction  construct  is 
based  on  the  existence  of  distinguishable  factors  which 
influence  the  construct.  A fundamental  prerequisite  to  the 
measurement  of  user  satisfaction  is  the  careful  identifi- 
cation of  the  factors  which  form  the  construct  domain. 

These  factors,  when  combined  with  the  semantic  differential 
methodology,  determine  "what"  is  to  be  measured  by  the 
concrete  measurement  actions. 

This  chapter  describes  the  methodology  for  identifying 
the  distinguishable  factors  which  form  the  domain  of  the 
user  satisfaction  construct.  A literature  review  is  pre- 
sented which  examines  previous  attempts  to  measure  user 
satisfaction.  The  literature  review  also  provides  the  basis 
for  establishing  the  initial  list  of  factors.  Operational 
definitions  and  a short  discussion  of  each  factor  to  be 
included  in  the  domain  are  given. 

These  factors  were  examined  by  an  independent  group  of 
expert  judges  as  a first  step  to  determine  the  completeness 
of  the  specified  factor  list.  The  results  of  this  examina- 
tion and  the  expanded  list  of  factors  that  evolved  is 
reported.  The  expanded  list  of  factors  was  then  subjected 
to  an  empirical  test  to  determine  the  completeness  of  the 
list.  Data  collection  for  the  test  required  an  extensive 
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interviewing  and  data  extraction  methodology  which  is 
described.  Finally,  the  empirical  test  results  are  pre- 
sented to  firmly  establish  the  factors  which  comprise  the 
domain  of  the  user  satisfaction  construct. 


LITERATURE  REVIEW 

Many  writers  have  considered  user  satisfaction  impor- 
tant; yet,  few  have  attempted  to  measure  it.  The  principal 
literature  that  was  reviewed  for  this  phase  of  the  research 
was  primarily  concerned  with  the  effectiveness  of  the  EDP 
function  within  organizational  settings  as  an  influence  on 
an  individual  user's  satisfaction.  The  review  focused  on 
identifying  studies,  surveys,  reports,  or  interviews  that 
not  only  dealt  with  a specified  user  population,  but  also 
indicated  the  underlying  factors  in  the  user-computer  rela- 
tionship that  were  considered  important.  There  was,  unfor- 
tunately, a paucity  of  literature  that  directly  addressed 
user  satisfaction.  The  vast  majority  of  the  literature  that 
was  examined  was  oriented  toward  some  purpose  other  than 
user  satisfaction.  Studies  were  identified,  however,  which 
contributed  piecemeal  information  about  the  factors  which 
affect  a user's  satisfaction. 

These  studies  were  located  primarily  through  the  use  of 
bibliographical  sources  and  personal  examination  of  the 
prominent  publications  concentrating  on  EDP  subjects.  Computers 
in  the  Organization  was  published  by  the  Defense  Logistics 
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Studies  Information  Exchange  (1974)  and  contained  117 
entries.  The  second  annotated  bibliographical  source,  the 
Quarterly  Bibliography  of  Computers  and  Data  Processing 
(1974,  1975,  1976),  provided  periodic  abstracting  coverage 
of  100  journals  relevant  to  the  EDP  field.  Pertinent  books 
and  reports  were  also  included.  The  final  bibliographical 
source  by  McLean,  Mason,  and  Foote  (1975)  listed  526  entries 
on  the  behavioral  and  organizational  aspects  of  computers 
and  allied  technology. 

The  purpose  of  the  literature  review  was  two-fold: 

1.  To  identify  any  previous  attempts  to  measure  user 
satisfaction. 

2.  To  identify  factors  which  influence  a user's 
satisfaction  with  the  computer-based  information  products 
and  services  that  are  provided  within  the  organization. 

Three  studies  did  address  the  measurement  of  user 
satisfaction.  In  each  study,  this  effort  was  a peripheral 
task  rather  than  the  main  objective  of  the  research.  The 
first  of  these  studies  was  initiated  by  Powers  and  Dickson 
(1973).  The  research  objective  was  to  identify  factors 
which  were  correlated  with  successful  management  information 
systems  (MIS).  Thirty-four  factors  were  hypothesized  which 
could  possibly  affect  the  success  or  failure  of  MIS  pro- 
jects. These  factors  were  then  ranked  by  a group  of  expert 
MIS  professionals,  and  11  of  the  34  factors  were  selected 
for  a field  study.  Four  success  criteria  were  selected  to 
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test  the  investigators'  hypotheses.  The  criteria  of  success 
were  (Powers  and  Dickson,  1973:150): 

1.  Time  - actual  time  to  complete  the  project/time 
estimated  for  the  project. 


cost , 


2.  Cost  - actual  cost  to  develop  the  project/budgeted 


3.  User  satisfaction  - attitudes  of  managers  receiving 


project  products  relative  to  how  well  their  information 
needs  were  being  satisfied. 

4.  Computer  operations  - the  impact  of  the  project  on 
the  computer  operations  function. 

Data  was  collected  via  an  interview  process  in  ten 
different  companies.  After  comparing  the  relationship  of 
the  eleven  factors  to  the  four  success  criteria,  the  authors 
stated,  "With  respect  to  MIS  projects,  it  is  our  opinion 
that  the  user  satisfaction  criterion  is  the  most  critical 
one  of  the  four  posited  for  this  study"  (Powers  and  Dickson, 
1973:153).  A major  deficiency  of  the  study  was  the  lack  of 
a methodology  to  quantify  the  extent  of  user  satisfaction. 

Lucas  (1973a)  employed  a questionnaire  in  his  study  of 
the  relationship  between  users  and  the  information  services 
department  of  seven  companies.  He  found  a strong  relation- 
ship between  certain  management  and  system  design  practices 
of  the  information  services  department  and  the  attitudes  and 
perceptions  of  the  users.  Lucas  used  the  terms  "attitudes 
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and  perceptions"  in  a fashion  that  corresponded  to  the 
definition  of  "user  satisfaction". 

The  questionnaire  utilized  by  Lucas  to  quantify  the 
level  of  user  attitudes  and  perceptions  consisted  of  a 
series  of  scales  related  to  the  following  factors: 

1.  Attitudes  toward  the  EDP  staff 

2.  Ratings  of  computer  potential 

3.  Perceived  quality  of  input 

4.  Perceived  quality  of  output 

5.  Training  received 

6.  Involvement  reported 

7.  Perceived  management  support 

The  questionnaire  items  were  formulated  as  statement 
completion  responses.  Responses  could  be  varied  along  a 
seven  interval  continuum  between  a positive  and  negative 
statement.  Each  factor  score  was  determined  by  the  mean 
response  to  all  scales  related  to  a particular  factor. 

The  primary  shortcomings  of  the  questionnaire  used  by 
Lucas  stemmed  from  the  lack  of  completeness  and  validation 
of  the  instrument  itself.  The  actual  source  of  the  items 
selected  for  each  scale  was  not  stated.  If  any  validation 
of  the  questionnaire  was  performed,  the  methodology  and 
results  were  not  reported.  There  were  several  factors  which 
were  not  included  in  the  questionnaire  that  other  studies 
considered  to  be  important  (Colton,  1973;  Firnberg,  1973). 
These  criticisms  of  Lucas'  questionnaire  were  tempered  by 
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the  realization  that  his  research  objective  was  the 
relationship  between  other  factors  and  the  measurement 
values  of  the  user's  attitudes  and  perceptions,  not  the 
measurement  process  itself. 

A more  recent  attempt  to  measure  user  satisfaction  was 
conducted  by  Swanson  (1974)  in  his  study  of  the  relationship 
between  a user's  involvement  with  and  appreciation  for 
management  information  systems  (MIS) . The  term  "appreci- 
ation" was  described  in  the  following  statement: 

The  MIS  appreciation  of  a manager  consists  of  his 
manifold  of  beliefs  about  the  relative  value  of  the  MIS 
as  a means  of  inquiry.  A manager's  appreciation  of  the 
MIS  is  to  be  described  in  terms  of  his  assessment  of  the 
system  as  a means  to  his  own  ends.  Thus,  if  a manager 
believes  his  MIS  reports  to  be  "timely",  "informative", 
"easy  to  understand",  and  so  forth,  we  might  say  that  he 
appreciated  the  system's  reports  (Swanson,  1974:179). 

It  was  clear  from  the  above  statement  that  Swanson's 
use  of  the  term  "appreciation"  was  similar  to  the  term 
defined  by  "user  satisfaction".  Swanson,  like  Lucas 
(1973a),  utilized  a questionnaire  to  indicate  MIS  apprecia- 
tion. The  questionnaire  consisted  of  16  items  which 
required  statement  completion  responses.  The  responses 
could  be  varied  along  a five  interval  continuum  between  a 
very  positive  and  a very  negative  statement.  Each  interval 
was  assigned  a value  ranging  from  .1  to  .9.  The  index  of 
appreciation  was  defined  as  the  simple  average  of  the  16 
individual  items.  Swanson  made  two  assumptions  in  con- 
structing the  index: 


First,  it  is  assumed  that  the  items  chosen  are  an 
adequate  sample  from  the  universe  of  items  which  would 
indicate  the  relative  presence  or  absence  of  MIS 
appreciation.  Secondly,  it  is  assumed  that  the  items 
are  equivalent  indicators  and  that  they  should  be 
weighted  equally  in  the  scoring  (Swanson,  1974:183). 

The  first  assumption  may  or  may  not  be  valid.  Although 
the  results  of  Swanson's  study  were  based  upon  the  indexes 
constructed  from  the  questionnaire,  no  evidence  to  substan- 
tiate this  assumption  was  reported.  The  second  assumption 
of  equal  weighting  was  equally  unsubstantiated.  Results 
reported  in  this  document  show  that  users  do  not  consider 
different  factors  equally  important;  therefore,  the  equal 
weighting  of  factor  scores  could  distort  the  resulting  index 
measurement . 

Powers  and  Dickson  (1973),  Lucas  (1973a),  and  Swanson 
(1974)  utilized  three  different  means  to  measure  variables 
very  similar  to  the  variable  of  user  satisfaction.  Although 
deficiences  preclude  their  universal  adoption,  they  did 
offer  clues  to  what  factors  do  influence  user  satisfaction. 
Additional  literature  was  reviewed  to  formulate  a compre- 
hensive list  of  factors  that  other  researchers  considered 
important  to  user  satisfaction.  The  criterion  for  the 
studies  to  be  considered  was  that  the  study  or  article  be 
based  upon  results  of  interaction  with  users  or  user  groups. 
Nineteen  additional  studies  identified  factors  to  be 
included  on  the  list.  The  studies  are  summarized  in  Table 
3.1  by  year,  authors,  subjects,  methodology,  and  research 


purpose.  The  number  of  observations  or  subjects  in  each 
study  is  denoted  by  "n". 


Each  of  the  studies  summarized  in  Table  3.1  was 
reviewed  to  extract  those  factors,  either  explicitly  con- 
sidered in  the  research  or  revealed  by  user  comments,  that 
influenced  the  satisfaction  of  users  or  user  groups  with 
computer-based  information  products  and  services.  For 
convenience,  the  factors  were  grouped  into  five  categories 
expressed  by  the  components  of  the  model  illustrated  in 
Figure  3.1. 


Organizational  Context 


Figure  3 . 1 

Model  of  Factor  Categories 

This  model  denotes  a symbolic  representation  of  the 
relationship  of  the  user  and  the  information  products  and 
services  that  are  provided.  This  relationship  exists  within 
an  organizational  context.  The  source  of  the  product  or 
service,  is  expressed  in  the  model  as  the  EDP  center,  staff, 
and  policies.  The  interface  between  the  user  and  the 
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source,  and  the  product  or  service  itself  are  also  compo- 
nents. Finally,  the  user  has  internal  constructs  which 
influence  his  state  of  satisfaction.  These  constructs  are 
categorical  feelings  which  are  germane  to  each  individual 
user.  The  feelings  exist  on  the  basis  of  internal  standards 
which  form  the  user's  frame  of  reference  in  the  evaluation 
process . 

The  36  factors  which  were  identified  from  the  litera- 
ture cited  in  Table  3.1  are  listed  in  Table  3.2.  Opera- 
tional definitions  and  a short  discusssion  of  each  factor 
are  presented  in  the  next  section.  A matrix  of  the  factors 
and  the  different  studies  which  identified  them  is  illus- 
trated in  Appendix  3.1.  The  most  striking  characteristic  of 
this  matrix  was  the  relatively  small  number  of  factors 
mentioned  in  any  one  study.  The  factor  mentions  ranged  from 
1 to  16  with  an  average  of  7.5.  This  was  only  partially 
explained  by  the  different  research  objectives  that  are 
represented.  The  more  obvious  explanation  was  the 
heretofore  unarticulated  research  specifically  directed  at 
identifying  the  set  of  factors  that  influence  user 
satisfaction. 


Operational  Definitions  of  the  Factors 

An  operational  definition  was  formulated  for  each 
factor  that  was  identified  in  the  literature  review.  Some 
of  the  factors  were  relatively  simple  and  straightforward, 


Top  management  involvement 
Organizational  competition  with  EDP  unit 
Priorities  determination 

Charge-back  method  of  payment  for  services 


Relationship  with  the  EDP  staff 
Communication  with  the  EDP  staff 
Technical  competence  of  the  EDP  staff 
Attitude  of  the  EDP  staff 
Schedule  of  products  and  services 
Time  required  for  new  development 
Processing  of  change  requests 
Vendor  support 

Response/ turnaround  time 
Mode  of  interface 
Convenience  of  access 

Accuracy 

Timeliness 

Precision 

Reliability 

Currency 

Completeness 

Format  of  output 

Language 

Volume  of  output 
Relevancy 
Error  recovery 
Security  of  data 
Documentation 

Expectations 

Understanding  of  systems 
Perceived  utility 
Confidence  in  systems 
Feeling  of  participation 
Feeling  of  control 
Degree  of  training 
Job  effects 


Organizational 

Context 


EDP  Center, 
Staff,  and 
Policies 


Interface 


Quality 

of 

Systems 


User 

Constructs 
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while  others  reflected  the  aggregation  of  several  complex 
facets  of  a particular  aspect  of  the  computer-based  support. 
The  level  of  aggregation  attempted  to  subsume  these  differ- 
ent facets  without  a severe  loss  of  information.  The  dif- 
ferent perspectives  of  a complex  factor,  such  as  the  Rela- 
tionship with  the  EDP  staff,  were  influenced  by  an  extensive 
literature  review  that  encompassed  many  sources  beyond  those 
cited  in  Table  3.1. 

The  definitions  were  formulated  to  avoid  the  defini- 
tional problems  described  by  Jones  in  his  research  on 
organizational  factors: 

The  entities  of  the  organization  would,  no  matter  how 
coercively  real  they  might  appear,  tend  to  escape  a 
tight  exact  definition. . . Attempts  at  definition  often 
have  diffused  frustratingly  the  image  of  a concept  which 
had  appeared  previously  as  vivid  and  precise  (Jones, 
1964:10) . 

Consequently,  the  definitions  are  intended  to  apply  to 

a broad  spectrum  of  users  rather  than  to  conform  to  overly 

rigid  specificity.  A brief  discussion  is  also  provided  for 

each  factor  where  appropriate.  The  operational  definitions 

and  discussions  are  shown  below. 

Top  management  involvement : The  positive  or  negative 
degree  of  interest,  enthusiasm,  support,  or  participa- 
tion of  any  management  level  above  the  user's  own  level 
toward  computer-based  information  systems  or  services  or 
toward  the  computer  staff  which  supports  them. 

The  literature  consistently  extolled  the  virtue  of 


positively  involved  top  management  as  a stepping  stone  to 
"success"  in  the  implementation  and  utilization  of  computer 
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based  information  systems  (McKinsey  and  Company,  1969; 

Hofer,  1970;  Fitts,  1971;  Colton,  1973).  Involvement  is  an 
attribute  reflecting  the  degree  and  direction  of  the  man- 
agement's support  and  enthusiasm  for  or  against  computer- 
based  information  systems.  The  attribute  may  be  manifested 
in  numerous  ways,  such  as  the  relegation  of  total  responsi- 
bility to  a high  or  low  organizational  hierarchical  level, 
public  remarks  denegrating  or  praising  the  use  of  computers, 
personal  involvement  or  non- involvement  with  planning  and/or 
major  operational  decisions,  and  others. 

The  users  themselves  reflect  the  attitudes  of  top  man- 
agement according  to  Constant  (1973) . He  observed  in  one 
organization  that  during  two  periods  of  radically  different 
top  management  attitudes  and  involvement,  the  prevailing 
attitudes  at  the  top  levels  spread  throughout  the  organ- 
ization. 

Holmes  emphatically  stated: 

It  is  absolutely  certain  that  any  softening  in 
interest,  involvement  or  support  demonstrated  by  top 
management  will  flow  freely  down  through  all  segments  of 
the  organization  and  present  a major  handicap  to  the 
success  of  any  information  system  (Holmes,  1970:27). 

Organizational  competition  with  the  EDP  unit:  The 
contention  between  the  respondent's  organizational  unit 
and  the  EDP  unit  when  vying  for  organizational  resources 
or  for  responsiblity  for  success  or  failure  of  computer- 
based  information  systems  or  services  of  interest  to 
both  parties. 

In  an  extensive  survey  involving  577  respondents,  Stone 


and  Tarnowieski  reported: 


/ 
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There  is  some  evidence  in  our  study  that  both  system 
users  and  systems  builders  retain  a goodly  number  of 
belligerent  thoughts.  In  some  instances  they  may  do  so 
because  they  regard  each  other  primarily  as  competitors 
and  secondarily  as  collaborators-and  then  only  because 
» of  circumstance  (Stone  and  Tarnowieski,  1972:8). 

The  feelings  of  competition  between  the  user's  organi- 
zational unit  and  the  EDP  unit  can  spring  from  several 

* sources.  One  source  may  be  the  straightforward  competition 
for  the  organizational  resources  of  capital  and  manpower. 

It  is  not  untypical  for  the  budget  and  manpower  allocations 

* for  EDP  to  be  expanding  at  the  same  time  that  cutbacks  are 
being  forced  on  other  organizational  units  (Miller,  1971). 

Both  organizational  units  may  claim  credit  for  func- 
tional improvements  when  computer  systems  are  involved. 
Like-wise,  the  blame  for  the  failure  of  new  information 
systems  to  achieve  expectations  may  also  be  vigorously 
contested.  Competition  fostered  under  these  conditions  can 
lead  to  conflict  which  greatly  affects  the  user's  satis- 
faction. 

Priorities  determination:  Policies  and  procedures  which 
establish  precedence  for  the  allocation  of  EDP  resources 
and  services  between  different  organizational  units  and 
their  requests. 

The  policies  establishing  precedence  for  service  when 
demand  outstrips  capacity,  the  schedules  for  new  application 
development,  and  other  actions  affect  the  user's  perception 
of  the  priorities  associated  with  the  allocation  of  the  EDP 
resources.  The  determination  of  these  priorities  can  have  a 
significant  effect  on  the  user's  access  and  utilization  of 
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computer-based  information  services. 

This  factor  appears  to  have  a more  pronounced  effect 
when  the  organizational  differentiation  is  marked  by  clearly 
competitive  units  (Constant,  1973).  An  example  of  such  an 
organization  would  be  a university  vrhere  the  administrative 
functions  compete  with  the  academic  functions  for  a limited 
EDP  resource.  The  competition  between  the  R&D  functions 
requiring  rapid  turnaround  of  unique  applications  and  the 
regularly  scheduled  production  type  functions  in  an  organ- 
ization is  another  example. 

These  examples  illustrate  situations  whereby  the  user's 

perception  of  the  priorities  associated  with  his  demands  on 

the  EDP  resource  could  affect  his  satisfaction.  The  factor 

may  be  Interpreted  as  one  measure  of  the  user's  functional 

competitive  position  within  the  organization. 

Charge-back  method  of  payment  for  services:  The 
schedule  of  charges  and  the  procedures  for  assessing 
users  on  a pro  rate,  basis  for  the  EDP  resources  and 
services  that  they  utilize. 

The  arguments  for  and  against  methods  of  charging  users 
for  the  EDP  resources  on  a pro  rata  basis  have  been  explored 
by  several  investigators  (Dearden  and  Nolan,  1973;  Lucas, 
1973b) . Notwithstanding  the  merits  or  demerits  of  these 
arguments,  Lucas  (1973a)  did  find  a relationship  between  the 
user  attitudes  and  the  existence  or  absence  of  a charge-back 
method  of  payment  in  operation. 


Thus,  the  charge-back  method  can  affect  the  user's 


satisfaction  with  his  support.  The  user  may  perceive  the 


method  itself  to  be  unfair.  The  user  may  also  feel  con- 


strained to  operate  under  the  charge-back  system  without 


having  the  option  of  seeking  more  economical  service  else- 


where. A method  biased  in  favor  of  a particular  user  may 


have  an  opposite  effect.  This  factor  appears  to  affect  user 
satisfaction  only  when  the  charge-back  method  is  being  used. 


The  absence  of  such  a method  does  not  seem  to  have  an  effect 


(Lucas,  1973a). 


Relationship  with  the  EDP  staff:  The  manner  and  methods 
of  interaction,  conduct,  and  association  between  the 


user  and  the  EDP  staff. 


The  user's  concept  of  his  relationship  with  the  EDP 


staff  is  a complex  factor.  The  relationship  is  influenced 


by  a multitude  of  other  factors.  Fitts  suggested  some  of 


the  influences : 


Poor  relations  between  computer  personnel  and  functional 
departments  may  be  caused  by  (1)  failure  to  "sell"  the 
computer  role  in  the  organization,  (2)  computer  opera- 
tion oriented  on  computer  efficiency  rather  than 
customer  satisfaction,  (3)  failure  of  computer  personnel 
to  understand  organization  goals  and  priorities,  and  (4) 
poor  overall  computer  operaton  (Fitts,  1971:48). 


The  list  of  influences  cited  by  Fitts  (1971)  was  not 


complete.  Constant  (1973)  noted  that  the  computer  personnel 


in  one  company  were  regarded  with  great  hostility  because  of 
their  apparent  loyalty  to  the  computer  profession  above  the 
needs  and  requirements  of  the  organization.  The  relation- 
ship was  also  affected  by  the  attitudes  and  habits  of  the 
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computer  personnel  who  had  direct  contact  with  the  users. 

The  relationship  between  the  user  and  the  EDP  staff  is 
characterized  by  the  dependencies  which  are  created  by  the 
nature  of  the  relationship.  Lucas  (1973b: 29)  observed  that 
"the  relationship  between  users  and  the  information  services 
unit  is  asymmetric  because  the  information  services  staff 
often  feels  it  must  understand  users'  jobs  whereas  the 
reverse  is  not  true." 

In  their  model  of  interdepartmental  conflict,  Walton 
and  Dutton  (1969)  argued  that  such  an  asymmetric  dependency 
leads  to  conflict.  Conflict,  which  may  vary  in  intensity, 
could  influence  the  user-EDP  staff  relationship  and,  subse- 
quently, the  user's  satisfaction. 

Communication  with  the  EDP  staff:  The  manner  and 
methods  of  information  exchange  between  the  user  and  the 
EDP  staff. 

Communication  between  the  user  and  the  EDP  staff  is 
beset  with  many  obstacles.  One  of  the  primary  obstacles  is 
the  lack  of  a common  language  or  jargon  (Teichroew,  1971). 
The  EDP  environment  abounds  with  terms  which  have  meaning 
only  within  that  narrow  milieu.  Many  user  groups  likewise 
have  a jargon  designed  to  fit  their  job  environment. 

Rarely,  do  the  two  share  much  common  ground.  After  an 
intensive  study,  Teichroew  flatly  states: 

There  is  a lack  of  communication  between  the 
managers  and  staff  personnel,  between  those  who  are  the 
users  of  administrative  information  systems  and  those 
with  the  technical  knowledge  to  conceive  and  implement 
such  systems  (Teichroew,  1971:579). 
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Communication  is  also  hindered  by  the  lack  of  a concep- 
tual framework  which  can  be  understood  by  both  groups. 
Therefore,  users  find  it  difficult  to  express  their  needs 
explicitly.  This  frustrates  the  system  developers  who 
strive  to  "lock-in"  a system  design  before  they  begin  actual 
development.  In  a study  of  the  factors  correlating  to  MIS 
success,  Powers  and  Dickson  (1973)  found  evidence  that  an 
evolutionary  type  of  system  development  which  utilized  the 
user's  learning  process  in  his  own  information  decision 
situation  significantly  increased  the  user's  satisfaction. 
Thus,  the  degree  of  success  that  is  achieved  in  communi- 
cation between  the  user  and  the  EDP  staff  can  have  a 
material  effect  on  the  user's  satisfaction. 

Technical  competence  of  the  EDP  staff:  The  computer 
technology  skills  and  expertise  exhibited  by  the  EDP 
staff. 

Colton  (1973:94)  found  that  the  competence  of  the  EDP 
staff  had  a material  effect  on  the  success,  and  subsequent 
user  satisfaction,  of  computer-based  information  systems  in 
police  departments.  This  effect  was  refected  in  the  user's 
confidence  in  the  systems  produced  and  in  the  user's  pro- 
pensity for  risk-taking  in  the  developmental  process.  The 
lack  of  competence  can  lead  to  increased  user  frustration 
when  problems  cannot  be  resolved  rapidly  with  a minimum  of 
downtime. 


Attitude  of  the  EDP  staff:  The  willingness  and  commit- 
ment  of  the  EbP  staff  to  subjugate  external,  profes- 
sional goals  in  favor  of  organizationally  directed  goals 
and  tasks. 

The  perceived  attitude  of  the  EDP  staff  is  often  de- 
scribed as  being  "user-oriented"  or  "EDP-oriented" . This 
attitude  impression  is  often  expressed  by  users  in  terms  of 
organizational  loyalty.  As  previously  stated,  the  lack  of 
organizational  loyalty  can  lead  to  outright  hostility  be- 
tween users  and  the  EDP  staff  (Constant,  1973).  This  result 
occurs  because  the  users  view  the  EDP  staff  as  using  their 
position  and  competence  in  a misdirected  fashion.  The  user- 
oriented  staff,  however,  is  perceived  as  one  that  is 
empathetic  to  the  user  requirements  and  problems.  This 
evaluation  of  the  attitude  of  the  EDP  staff  can  affect  the 
user's  overall  satisfaction. 

Schedule  of  products  and  services:  The  EDP  center  time- 
table for  production  of  information  system  outputs  and 
for  provision  of  computer-based  services. 

This  factor  expresses  how  well  the  production  schedule 
of  the  EDP  center  corresponds  to  the  user  requirements.  One 
example  would  be  the  time  of  day  that  a system  report  is 
processed  contrasted  to  the  time  of  day  that  the  user  wants 
to  receive  the  report.  The  frequency  is  also  a considera- 
tion. A service  example  would  be  the  allocation  of  six 
hours  a day  for  on-line  processing  when  a user  desires  eight 
hours  a day  for  terminal  operations . 
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Time  required  for  new  development : The  elapsed  time 
between  the  user's  request  for- new  applications  and  the 
design,  development,  and/or  implementation  of  the 
applications  systems  by  the  EDP  staff. 

The  ability  of  the  EDP  staff  to  respond  to  new  develop- 
ment requests  can  have  quite  an  effect  on  the  user's  satis- 
faction (Neuschel,  1971;  Stone  and  Tarnowieski,  1972; 

Colton,  1973) . In  a survey  of  220  general  managers  (users) , 
72  respondents  considered  consistent  "on  time"  development  a 
primary  criterion  for  evaluation  of  the  EDP  function  (Stone 
and  Tarnowieski,  1972). 

This  factor  is  heavily  influenced  by  the  user's 

expectations.  It  is  subject  to  misconceptions  about  the 

ease  or  difficulty  of  the  technical  problems  involved  in 

developing  new  systems.  The  reasonableness  of  the  time 

required  for  new  development  is  linked  to  the  individual 

user's  internal  judgmental  structure. 

Processing  of  change  requests:  The  manner,  method,  and 
required  time  with  which  the  EDP  staff  responds  to  user 
requests  for  changes  in  existing  computer-based  infor- 
mation systems  or  services. 

As  one  user  stated,  "I  want  to  be  able  to  make  changes 
as  the  world  changes"  (Constant,  1973).  Fitts  (1971)  found 
unresponsiveness  to  change  requests  a major  irritant  to 
users  in  his  study.  The  explanation  may  be  the  rising 
expectations  of  users  as  their  contact  with  EDP  technology 
increases,  as  well  as  changes  in  their  job  environment 
(Lucas,  1973a). 
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Vendor  support : The  type  and  quality  of  the  service 
rendered  by  a vendor,  either  directly  or  indirectly,  to 
the  user  to  maintain  the  hardware  or  software  required 
by  that  user.  A vendor  is  distinguished  by  his  external 
organizational  status. 

The  increased  use  of  terminals  by  users  and  the  growth 

of  dedicated  mini-computers  for  functional  operations  has 

increased  the  direct  user-vendor  contact  (Hobbs  and 

McLaughlin,  1974) . This  contact  is  also  increased  by  the 

use  of  purchased  software  packages  which  are  maintained  by 

the  vendor.  This  relationship  is  characterized  by  the  same 

dependencies  expressed  in  the  user-EDP  staff  relationship. 

Response/ turnaround  time:  The  elapsed  time  between  a 
user-initiated  request  for  service  or  action  and  a reply 
to  that  request.  Response  time  generally  refers  to  the 
elapsed  time  for  a terminal  type  request  or  entry. 
Turnaround  time  generally  refers  to  the  elapsed  time  for 
execution  of  a program  submitted  or  requested  by  a user 
and  the  return  of  the  output  to  that  user. 

In  a case  study  of  two  companies  with  a heavy  emphasis 
on  terminal  usage,  Firnberg  (1973:132)  noted,  "The  users 
found  the  5-10  second  response  time  satisfactory,  the 
occasional  20  seconds  were  frustrating".  Colton  (1973) 
found  that  response  time  was  even  more  critical  to  some 
applications  users  in  police  departments. 

In  a study  of  the  behavioral  factors  in  information 
systems,  Testa  reported: 

An  examination  of  these  time-sharing  systems,  however, 
reveals  a pervasive  failure  to  consider  human  behavioral 
capabilities.  For  example,  the  frequent  occurrence  of 
response  delays  in  time-sharing  systems  can  be  dis- 
ruptive of  problem  solving...  delays  of  approximately  15 
seconds  cause  deteriorated  performance  in  problem 
solving  situations  (Testa,  1974:16). 
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Turnaround  time  was  a major  source  of  dissatisfaction 
in  Constant's  (1973)  study.  He  discovered  that  delayed 
turnaround  of  specialized  engineering  application  programs 
was  responsible  for  work  stopages.  Progress  on  the  engi- 
neering projects  could  not  proceed  until  the  output  was 
received  from  the  EDP  center.  The  lack  of  rapid  turnaround 
created  great  hostility  between  the  two  departments. 

These  limited  examples  highlight  the  effect  of 
response/ turnaround  time  on  the  user's  satisfaction. 

Although  the  two  aspects  are  different,  they  are  combined 
into  one  factor.  This  combination  reflects  the  tendencies 
of  users  to  be  concerned  primarily  with  either  response  or 
turnaround  time,  not  both. 

Mode  of  interface:  The  method  and  medium  by  which  a 
user  inputs  data  to  and  receives  output  from  the  EDP 
center. 

This  factor  is  defined  by  the  means  by  which  the  user 
inputs  data  to  and  receives  output  from  the  EDP  center. 

Thus,  a user's  mode  of  interface  may  be  a deck  of  data  cards 
submitted  by  mail  carrier  and  the  printed  hard  copy  received 
after  processsing  via  the  same  carrier.  Another  mode  of 
interface  may  be  a single  terminal  for  both  transmission  and 
receipt  of  information.  The  many  possible  combinations  all 
vary  in  the  convenience,  speed,  cost,  etc.  for  the  user. 

The  effect  of  the  mode  of  interface  on  the  user  was 
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discussed  by  several  authors.  Lucas  (1973a)  found  that  on- 
line systems  were  associated  with  more  favorable  user 
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attitudes.  Testa  (1974)  argued  that  different  cognitive 

types  of  users  were  affected  differentially  by  the  mode  of 

interface.  Yet,  a matching  of  the  most  effective  mode  to 

individual  users  was  still  in  the  experimental  stage  (Mason 

and  Mitroff,  1973).  This  factor,  therefore,  varied  in  its 

influence  on  the  user's  satisfaction. 

Convenience  of  access:  The  ease  or  difficulty  with 
which  the  user  may  act  to  utilize  the  capablity  of  the 
computer  systems. 

Before  the  potency  of  computer  systems  can  be  gainfully 
employed,  the  user  must  have  access  to  that  capability.  The 
convenience  of  that  access  can  influence  the  user's  satis- 
faction. This  convenience  is  affected  by  a multitude  of 
factors.  The  location  and  number  of  terminals,  the  mechan- 
ical condition  of  keypunch  machines,  the  policies  and 
procedures  for  requesting  new  application  development,  or 
the  open  or  closed  status  of  the  EDP  center  itself  are 
examples  of  factors  which  affect  the  user's  accessibility 
(Firnberg,  1973). 

Accuracy:  The  correctness  of  the  output  information. 

Timeliness : The  availability  of  the  output  information 
at  a time  suitable  for  its  use. 

Precision:  The  variability  of  the  output  information 
from  that  which  it  purports  to  measure. 

Reliability:  The  consistency  and  dependability  of  the 
output  information . 

Currency:  The  age  of  the  output  information. 

Completeness : The  comprehensiveness  of  the  output 
information content . 
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The  quality  of  the  systems  that  support  the  user  was 
often  mentioned  in  the  literature  as  a source  of  satis- 
faction or  dissatisfaction.  The  preceeding  attributes  were 
cited  as  the  primary  indicators  of  quality.  They  appeared 
to  be  unidimensional  in  character.  Some  of  the  literature 
sources  for  each  attribute  are  shown  in  parenthesis: 

1.  Accuracy  (Stone  and  Tarnowieski,  1972) 

2.  Timeliness  (Swanson,  1974) 

3.  Precision  (Adams  and  Schroeder,  1973) 

4.  Reliability  (Adams,  1975) 

5.  Currency  (Lucas,  1973a) 

6.  Completeness  (Schussel,  1974) 

Adams  and  Schroeder  (1973)  found  evidence  that 
different  users  tend  to  discuss  different  attributes  with 
varying  intensity.  They  stated: 

One  interesting  phenomenon. . . noticed  in  talking 
with  middle  managers  is  their  tendency  to  concentrate  on 
one  characteristic  of  the  information  that  they  receive 
to  the  exclusion  of  all  others  (Adams  and  Schroeder, 
1973:65). 

The  multidimensional  nature  of  the  systems  quality  and 
the  tendency  of  users  to  consider  the  dimensions  differen- 
tially dictates  that  measurement  of  these  factors  provide 
for  user  weighting  of  each  attribute.  The  weighting  should 
reflect  the  differential  effect  of  the  attributes  on  the 
user's  overall  satisfaction. 

Format  of  output:  The  material  design  of  the  layout  and 
display  of  the  output  contents. 

Adams  and  Schroeder  (1973)  reported  that  20  percent  of 
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the  managers  they  interviewed  faulted  the  format  for  the 
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lack  of  desired  data.  The  format  of  the  output  can  facili- 
tate or  hinder  the  user's  utilization  of  the  information 
system,  and  hence,  his  satisfaction. 

The  attributes  of  clarity  and  summarization  of  the 
output  products  are  included  in  this  factor.  Clarity 
reflects  the  ease  of  user  comprehension  of  the  output 
contents.  Summarization  reflects  the  levels  of  aggregation 
of  data  elements  and  their  display.  Both  attributes  are 
combined  into  the  format  factor  as  it  affects  user  satis- 
faction. 

Language : The  set  of  vocabulary,  syntax,  and 

grammatical  rules  used  to  interact  with  the  ct  iputer 
systems . 

The  increased  use  of  terminals,  minicomputers,  and 
distributed  processing  has  fostered  an  increase  in  direct 
user-computer  interface  (Gruenberger  and  Babcock,  1973) . 

The  language  that  is  used  to  facilitate  this  interaction  has 
an  effect  on  the  user's  satisfaction.  Language  refers  to 
the  key  words  and  grammatical  rules  which  allow  the  user  to 
achieve  his  purpose.  That  purpose  may  be  a database  query, 
record  update,  data  entry,  or  any  of  the  others  allowed 
under  the  capabilities  of  the  particular  system  provided. 

The  user's  satisfaction  is  affected  by  the  clarity, 
versatility,  and  suitability  of  the  language.  Although  this 
factor  is  most  prominently  mentioned  in  conjunction  with  on- 
line applications,  it  is  not  restricted  to  that  situation. 
Language  can  also  be  an  influence  in  batch- type  applications 
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if  the  user  is  constrained  to  fixed  grainmers  for  utilization 
of  his  information  systems. 

Volume  of  output:  The  amount  of  information  conveyed  to 
a user  from  computer-based  systems.  This  is  expressed 
by  not  only  the  number  of  reports  or  outputs  but  also  by 
the  voluminousness  of  the  output  contents. 

Lucas  (1974)  found  a strong  negative  correlation 

between  the  number  of  reports  received  and  user  attitudes. 

This  sentiment  was  echoed  by  other  authors  (Fitts,  1971; 

Adams  and  Schroeder,  1973).  Holmes  stated: 

It  is  unbelievable  under  present-day  concepts  of 
exception  reporting  to  find  stacks  of  machine  runoffs 
being  delivered  to  senior  executives.  Sometimes  the 
user  finds  the  appropriate  pages  to  keep,  but  more  often 
all  is  discarded  (Holmes,  1970:33). 

The  mechanisms  for  handling  information  overload  were 

catalogued  by  Miller  (1960) : 

1.  Omission  - the  temporary  non-processing  of 
information. 

2.  Error  - the  processing  of  information  incorrectly. 

3.  Queuing  - the  delaying  of  a response  to  catch  up 
during  a "lull". 

4.  Filtering  - the  neglecting  of  certain  categories  of 
information  while  processing  others. 

5.  Cutting  categories  of  discrimination  - the  respond- 
ing to  input  in  a more  general  way  with  less  precision  than 
before  at  slower  rates. 
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6.  Employing  multiple  channels  - the  processing  of 
information  through  multiple  channels,  as  demonstrated  by 
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decentralize ton. 

7.  Escape  from  the  task  - the  avoiding  of  processing 
any  of  the  information. 

A user  who  is  forced  to  resort  to  one  of  these  re- 
sponses has  reason  to  be  dissatisfied  with  the  information 
systems  support.  This  factor  not  only  includes  the  volume 
as  a function  of  the  number  of  reports,  but  also,  as  a 
function  of  the  data  or  information  included  in  a single 
output. 

Relevancy:  The  degree  of  congruence  between  what  the 

user  wants  or  requires  and  what  is  provided  by  the 
information  products  and  services . 

Gupta  (1974)  surveyed  150  large  companies  and  dis- 
covered that  not  one  of  respondents  was  pleased  with  the 
development  of  their  information  systems.  One  of  the 
primary  causes  of  this  dissatisfaction  was  the  lack  of 
relevancy  of  the  output  products  to  the  concerns  of  the 
users . 

In  one  sense,  the  relevancy  of  the  information  products 
and  services  provided  is  the  crux  of  the  effectiveness  of 
the  support  rendered.  Users  cannot  and  will  not  use  infor- 
mation that  is  not  relevant  to  their  functional  require- 
ments. This  does  not,  however,  preclude  them  from  receiving 
irrelevant  informaton. 

Error  recovery:  The  methods  and  policies  governing 
correction  and  rerun  of  system  outputs  that  are 
incorrect. 
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Firnberg  (1973)  found  that  the  ease  of  correcting 
errors  was  a source  of  user  satisfaction  in  the  case  studies 
that  he  observed.  This  factor  may  be  closely  related  to  the 
Language  factor  and  the  scheduling  policies  for  the  EDP 
center.  Systems  which  are  subject  to  many  source  data 
errors  coupled  with  stringent  accuracy  requirements,  such  as 
some  accounting/finance  functions,  can  be  extremely  frus- 
trating to  the  user  unless  liberal  rerun  policies  allow 
rapid  error  correction  and  output  consolidation.  Automated 
financial  statements  which  require  a zero-balance  output  is 
one  example  whereby  the  ease  of  error  recovery  can  facili- 
tate user  satisfaction. 

Security  of  data:  The  safeguarding  of  data  from  mis- 
appropriation or  unauthorized  alteration  or  loss. 

The  sensitivity  of  the  data  may  lead  to  user  concern 
over  the  security  of  the  data.  Holland  (1974)  discovered 
that  many  users  specified  very  narrow  system  designs  to 
protect  the  accessibility  of  the  data  in  their  information 
systems.  Firnberg  (1973)  found  that  the  use  of  passwords 
and  security  codes  to  control  access  to  data  bases  increased 
user  satisfaction. 

Security  of  data  may  also  involve  the  physical  security 
afforded  by  the  EDP  center  itself.  Theft,  vandalism,  and 
natural  disasters,  such  as  floods  or  fires,  have  been  an 
increasing  concern  to  some  users  (Parker,  Nycum,  and  Oura, 
1973) . This  concern  has  been  reflected  in  the  fortress  type 
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design  of  some  EDP  centers  and  extensive  backup  facilities 
to  ensure  adequate  recovery  if  accidents  do  occur. 

The  user's  satisfaction  is  affected  by  the  security  of 
data,  both  in  a system  design  sense  and  a physical  safeguard 
sense.  This  factor  becomes  increasingly  important  as  the 
sensitivity  of  the  data  increases,  and  users  have  their  own 
different  concepts  of  what  constitutes  sensitivity  (Holland, 
1974) . 

Documentation : The  recorded  description  of  an  infor- 

mation system.  This  includes  formal  instructions  for 
e utilization  of  the  system. 

e documentation  of  an  information  system  may  affect 
user  in  two  ways.  It  may  first  influence  the  user's 
understanding  of  the  system  and  facilitiate  the  utilization 
of  that  system.  Secondly,  the  documentation  may  affect  the 
actual  processing  of  the  system  via  the  operating  instruc- 
tions. Failure  to  maintain  the  documentation  properly  and 
to  reflect  system  changes  can  result  in  the  incorrect 
execution  of  the  system. 

The  existing  documentation  is  often  the  initial  basis 

for  discussions  between  users  and  the  EDP  staff  for  new  or 

improved  system  development  (Smith  and  Wechsler,  1973).  The 

quality  of  that  documentation  is  an  important  source  of  user 

satisfaction  or  dissatisfaction. 

Expectations : The  set  of  attributes  or  features  of  the 
computer-based^ information  products  or  services  that  a 
user  considers"  reasonable  and  due  from  the  computer- 
based  information  support  rendered  within  his  organ- 
ization. 


Every  user  develops  expectations  about  the  capabilities 
engendered  by  computer  technology.  These  expectations  are 
altered  in  many  ways  by  many  sources:  the  user's  col- 
leagues, publications,  vendor  sales  presentations,  and  the 
user's  increasing  experience  and  learning  process  are  a few 
of  the  important  influences.  These  expectations  are 
satisfied  or  dissatisfied  to  a varying  degree. 

Dissatisfaction  is  often  registered  by  users  who  claim 
that  the  computer  capabilities  were  "oversold".  The  claim 
that  the  capabilities  of  the  systems  requested  fell  short  of 
those  advertised  while  the  costs  exceeded  the  budgeted 
amounts  was  repeated  with  regularity  in  the  literature 
(Neuschel,  1971;  Colton,  1973). 

This  factor  may  prove  to  be  quite  volatile  as  an 

influence  on  user  satisfaction.  Expectations  may  rise  or 

fall  rapidly  with  the  occurrence  of  single  events . One 

example  would  be  a user’s  discovery  of  the  existence  of  a 

powerful  information  system  in  a competitor's  organization 

that  is  not  provided  in  the  user's  own  organization. 

Another  example  would  be  a user's  introduction  to  the 

realities  of  the  demands  of  software  development.  The  user 

expectations  may  fall  as  the  user  learns  of  the  time 

required  for  complex  systems,  yet  rise  with  exposure  to  the 

capabilities  which  it  affords. 

Understanding  of  systems:  The  degree  of  comprehension 
that  a user  possesses  about  the  computer-based  infor- 
mations systems  or  services  that  are  provided. 
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An  increased  understanding  appears  to  generally  lead  to 

increased  satisfaction  for  the  user  (Colton,  1973).  The 

reverse  may  be  true  if  the  information  system  is  of  low 

quality  with  severe  shortcomings . One  reason  for  the  effect 

of  this  factor  may  be  that  increased  understanding  reduces 

the  user's  feeling  of  dependency  on  the  EDP  staff  and  their 

knowledge  of  the  system.  This  conjecture,  however,  has  no 

quantitative  support  at  this  time. 

Perceived  utility:  The  user's  judgment  about  the 
relative  balance  between  the  cost  and  the  considered 
usefulness  of  the  computer-based  informaton  products  or 
services  that  are  provided.  The  costs  include  any  costs 
related  to  providing  the  resource,  including  money, 
time,  manpower,  and  opportunity.  The  usefulness 
includes  any  benefits  that  the  user  believes  to  be 
derived  from  the  support. 

Dearden  and  Nolan  (1973:76)  stated,  "In  our  opinion, 
users  should  pay  a fair  price  for  the  computer  resources 
that  they  use.  After  all,  the  resource  should  serve  a 
purely  economic  function  for  the  user."  Regardless  of 
whether  a direct  method  of  payment  was  used  (charge-back)  or 
not,  most  writers  agreed  that  users  have  an  opinion  about 
the  utility  of  the  information  services  they  receive  (Fitts, 
1971;  Swanson,  1974;  Adams,  1975).  In  his  study  of  police 
departments,  Colton  (1973:95)  found  that  "the  computer  is 
viewed  by  some  departments  as  being  a nice  but  disposable 
luxury,  and  there  is  always  the  potential  for  waste." 


The  user's  perception  of  the  utility  of  the  support 
carries  beyond  a sheer  economic  cost  analysis.  In  some 
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functional  applications,  computer  systems  are  considered 
vital  to  the  success  and  even  the  existence  of  the  organi- 
zation. An  example  would  be  a banking  application  requiring 
massive  data  handling  within  severe  time  constraints  to  meet 
government  regulations  to  maintain  the  btnk's  charter.  Such 
utility  perceptions  influence  the  user's  satisfaction  with 
the  information  system  support. 

Confidence  in  the  systems:  The  user's  feelings  of 
assurance  or  certainty  about  the  systems  provided. 

The  user's  confidence  in  systems  is  applicable  during 
both  the  developmental  process  and  the  system  opera. ion. 

The  former  signifies  a belief  that  the  EDP  staff  can  and 
will  deliver  the  system  capability  as  agreed  upon.  Hie 
latter  involves  the  quality  and  reliability  of  service  of 
existing  systems.  Neuschel  (1971:85)  asserted,  "Mo  computer 
program  will  long  be  successful  if  there  is  no  credibility 
and  confidence  between  management  and  the  systems  and 
computer  people." 

Feeling  of  participation:  The  degree  of  involvement  and 
commitment  which  the  user  shares  with  the  EDP  staff  and 
others  toward  the  functioning  of  the  computer-based 
information  systems  and  services. 

A majority  of  the  managers  interviewed  by  Holland 
(1974)  felt  that  users  should  be  highly  involved  in  the 
system  design  process.  Tomeski  (1974)  elaborated  upon  that 
point  in  his  study  of  job  enrichment  and  the  computer. 

Other  writers  agreed  on  the  importance  of  participation 
(Dickson  and  Simmons,  1970;  Holmes,  1970).  This  support  for 
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user  involvement  was  touted  largely  for  its  effect  on  the 
"success"  of  the  resultant  system.  Yet,  Tomeski  (1974) 
effectively  argued  for  the  participation  because  of  its 
effect  on  user  satisfaction  as  well. 

Participation,  or  lack  of  it,  can  have  an  effect  on  the 

user's  understanding  and  confidence  in  the  system.  This 

factor  can  also  influence  the  user's  feeling  of  achievement, 

an  important  reinforcer  of  positive  behavior.  Both  effects 

are  registered  in  the  user's  satisfaction. 

Feeling  of  control:  The  user's  awareness  of  the 
personal  power  or  lack  of  power  to  regulate,  direct  or 
dominate  the  development,  alteration,  and/or  execution 
of  the  computer-based  information  systems  or  services 
which  serve  the  user's  perceived  function. 

The  importance  of  this  factor  was  most  strongly  stated 
by  Adams  (1975:343)  in  the  form  of  an  hypothesis  that 
"control  over  systems  and  systems  development  is  a major 
factor  in  the  satisfaction  of  management  information  systems 
users."  This  hypothesis  was  supported  by  data  collected  by 
Adams,  as  well  as  others  (Diebold,  1969;  Tatham,  1969; 

Fitts,  1971).  It  was  considered  a strong  element  of  user 
satisfaction. 

The  user's  feeling  of  control  has  several  facets.  It 
may  concern  budget  control  for  new  development,  design 
change/ initiation  decision  authority,  scheduling  control,  or 
others.  This  factor  may  also  be  affected  by  organizational 
considerations.  The  degree  of  centralization  of  the  EDP 
resources  and  staff,  coupled  with  the  EDP  organizational 
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position,  can  modify  the  user's  feeling  of  control. 

Degree  of  training:  The  amount  of  specialized  instruc- 
tion and  practice  that  is  afforded  to  the  user  to 
increase  the  user's  proficiency  in  utilizing  the 
computer  capability  that  is  available. 

A survey  conducted  by  Lucas  (1973a)  contained  a pro- 
vision for  open-ended  comments  in  the  questionnaire  on  user 
attitudes.  The  second  most  common  comment  from  the  respond- 
ents was  a desire  for  more  training.  The  data  also  reflect- 
ed a positive  relationship  between  the  amount  of  training 
and  favorable  attitudes  among  users.  One  reason  for  this 
relationship  could  have  been  the  more  realistic  expectations 
coupled  with  increased  understanding  of  the  systems  that  one 
would  expect  to  follow  additional  user  training. 

Job  effects : The  changes  in  job  freedom  and  job 
performance  that  are  ascertained  by  the  user  as 
resulting  from  modifications  induced  by  the  computer- 
based  information  systems  and  services. 

Two  primary  job  effects  induced  by  the  use  of  computer- 
based  information  systems  were  cited  in  the  literature 
(Holmes,  1970).  The  first  was  the  effect  on  the  user's 
s^nse  of  job  freedom.  One  user  could  feel  that  the  imposi- 
tion of  a structured  information  system  was  very  restric- 
tive. ^et,  another  user  could  feel  that  the  use  of  an 
information  system  offered  freedom  from  mundane,  repetitive 
tasks  and  allowed  concentration  on  the  more  interesting  and 
unstructured  aspects  of  the  job. 

The  second  job  effect  noted  was  the  effect  on  the 
user's  job  performance.  One  consistent  justification  for 
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implementing  computer-based  information  systems  was  to 
enable  users  to  perform  their  jobs  and  make  decisions  more 
rapidly  and  more  comprehensively  (Lucas,  1974).  These 
claims  were  justified  for  some  users.  Other  users  found 
that  the  positive  effects  on  their  job  performance  were 

insignificant  in  relation  to  the  requirements  imposed  by  the 

/ 

system.  This  resulted  in  a negative  net  effect  on  the 
user's  job  performance. 

Independent  Review 

The  literature  review  provided  an  extensive  list  of 
factors  that  influence  a user's  satisfaction.  As  a first 
step  in  validating  the  completeness  of  the  list,  the  factors 
and  their  operational  definitions  were  provided  to  three 
independent  judges  for  review.  The  purpose  of  this  review 
was  to  provide  an  independent  assessment  of  the  completeness 
of  the  list  of  factors.  The  collective  experience  of  the 
judges  augmented  the  literature  review  process  and  provided 
different  perspectives  to  counterbalance  any  pertinent 
omissions  that  could  have  occurred. 

The  judges  were  selected  on  the  basis  of  their  exper- 
ience in  the  data  processing  field  and  their  history  of 
formalized  interaction  with  users  and  user  groups.  The 
first  judge  was  the  manager  of  a commercial  data  processing 
service  bureau.  The  second  judge  was  a university  professor 
with  a varied  background  which  included  extensive  experience 
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as  a consultant  for  data  processing  and  organizational  prob- 
lems. The  third  judge  was  the  manager  of  a major  univer- 
sity's data  processing  center.  He  had  previous  experience 
in  a similar  position  with  a large  industrial  organization. 
The  third  judge  had  two  of  his  assistants  aid  him  in  the 
review  process;  therefore,  a total  of  five  persons  actually 
reviewed  the  list  of  factors. 

Each  judge  was  asked  to  review  the  list  of  factors  and 
the  definitions  and  to  comment  on  any  omissions  or  defini- 
tional shortcomings.  These  reviews  were  conducted  in  an 
unstructured  format.  Collectively,  the  judges  considered 
the  list  to  be  more  extensive  than  expected  and  expressed 
difficulty  in  finding  omissions.  Notwithstanding  the 
favorable  reviews  by  all  of  the  judges,  two  factors  were 
suggested  for  'nclusion  on  the  list.  These  factors  were 
added  to  the  ' t and  are  shown  below  with  their  operational 
definitions : 

Organiza  ,nal  Position  of  the  EDP  Function:  The 
hierarchical  relationship  of  tbe  EDF  function  to  the 
overall  organizational  structure. 

Flexibility  of  Systems:  The  capacity  of  the  information 
system  to  change  or  to  adjust  in  response  to  new  condi- 
tions, demands,  or  circumstances. 

The  suggested  additions  and  the  independent  assessment 
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of  the  completeness  of  the  factor  list  indicated  that  the 
important  factors  that  influence  a user's  satisfaction  with 
computer-based  information  products  and  services  had  been 
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identified.  An  empirical  test  of  this  concept  and  a formal- 
ized definition  of  importance  was  then  designed  and  conduct- 
ed to  further  validate  the  claim  of  completeness.  The  test 
design  and  test  results  appear  in  the  following  sections. 


'* 


EMPIRICAL  TEST  OF  THE  FACTOR  LIST 
The  factor  list  illustrated  in  Table  3.3  was  compiled 
to  establish  the  fundamental  dimensions  for  measuring  user 
satisfaction.  The  objective  of  this  phase  of  the  research 
was  to  test  the  completeness  of  the  list  of  factors.  A 
formal  restatement  of  this  objective  is  developed  below. 

Let  U denote  the  universal  set  of  all  factors  which 
influence  a user's  satisfaction.  The  set  U is  assumed  to  be 
finite  but  unknown.  Let  F denote  the  subset  of  U expressed 
by  the  factors  in  Table  3.3.  The  element  x signifies  any 
factor  that  a randomly  selected  user  considers  important  as 
an  influence  on  his  state  of  satisfaction.  The  claim  that  F 
is  a sufficiently  complete  list  of  important  factors  is 
determined  by  the  probability  of  x being  a member  of  F. 
Specifically,  the  subset  F is  defined  to  be  sufficiently 
complete  if  the  probability  of  x being  an  element  of  F is 
greater  than  .90,  which  may  be  restated  as: 

Pr  (x  e F)  > .90  (3.1) 

This  definition  is  based  on  the  rationale  that  impor- 


tant factors  tend  to  be  shared  by  many  users.  Although  each 
individual  user  may  specify  a unique  set  of  factors  which 


Table  3.3 

Factors  Affecting  User  Satisfaction 


Factor 

Top  management  involvement 
Organizational  competition  with  EDP  unit 
Priorities  determination 

Charge-back  method  of  payment  for  services 
Organizational  position  of  the  EDP  function 

Relationship  with  the  EDP  staff 
Communication  with  the  EDP  staff 
Technical  competence  of  the  EDP  staff 
Attitude  of  the  EDP  staff 
Schedule  of  products  and  services 
Time  required  for  new  development 
Processing  of  change  requests 
Vendor  support 

Response/ turnaround  time 
Mode  of  interface 
Convenience  of  access 


Category 

Organizational 

Context 


EDP  Center 
Staff,  and 
Policies 


Interface 


Accuracy 

Timeliness 

Precision 

Reliability 

Currency 

Completeness 

Flexibility 

Format  of  output 

Language 

Volume  of  output 

Relevancy 

Error  recovery 

Security  of  data 

Documentation 


Quality 

of 

Systems 


Expectations 

Understanding  of  systems 
Perceived  utility 
Confidence  in  systems 
Feeling  of  participation 
Feeling  of  control 
Degree  of  training 
Job  effects 


User 

Constructs 
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I^Sfe******  ti-**  ~ • 


* 

ll 


» 


^T-T7'  '•  ■ ' 


/ 


influence  satisfaction,  the  important  factors  will  be  cited 


with  greater  frequency  over  the  entire  population  of  users, 


If  the  probability  condition  of  Equation  3.1  is  true,  then 


at  least  90  pe. 


c the  factors  which  affect  the  satis- 


faction ef  any 


selected  user  is  expected  to  be  con- 


tained in  the  set  1 


nsequently,  the  set  F forms  a suffi- 


cient domain  for  measuring  the  critical  content  of  any 


user's  satisfaction  construct. 


The  completeness  of  F may  be  verified  by  a test  of  the 


probability  of  any  factor  x being  included  in  F.  The  iden- 


tification of  a factor  by  a user  is  an  event  which  can  have 


one  of  two  mutually  exclusive  outcomes.  The  factor  is 


either  a member  of  F,  or  it  is  not.  Therefore,  each  factor 


identified  by  a user  may  be  regarded  as  an  independent 


Bernoulli  trial. 


A test  based  on  the  binomial  distribution  was  used  to 


evaluate  the  completeness  of  F.  The  test  was  conducted  in 


the  following  manner: 


1.  The  set  F was  defined  by  the  factors  in  Table  3.3. 

2.  A representative  sample  of  users  was  selected. 


3.  Each  user  was  subjected  to  an  interview  process 


designed  to  elicit  the  factors  which  influence  the  user's 


satisfaction. 


:4.  Each  factor  that  was  mentioned  was  extracted  from 


the  interview. 


I 


— 
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5.  Each  factor  was  labeled  a success  (was  an  element 
of  F)  or  a failure  (was  not  an  element  of  F) . 

6.  Each  mention  of  a factor  was  regarded  as  an 
independent  trial. 

7.  The  test  statistic  was  calculated  from  the  fre- 
quencies of  successes  and  failures  based  upon  a binomial 
distribution  with  a probability  of  success  of  greater  than 
.90. 

The  null  hypothesis  for  this  test  was: 

H0  : Pr(x  e F)  < .90 

The  alternative  hypothesis,  which  could  be  accepted  if 
H was  rejected,  was: 

Hi  : Pr (x  e F)  > .90 

The  use  of  a test  based  on  a binomial  distribution  must 
satisfy  four  assumptions  (Bradley,  1968:166): 

1.  Outcomes  must  be  capable  of  being  dichotomized. 

2.  The  two  outcomes  must  be  mutually  exclusive. 

3.  The  outcome  of  the  events  must  be  independent. 

This  means  that  the  probability  of  "success"  must  not  change 
from  trial  to  trial. 

4.  Events  must  be  randomly  selected. 

The  first  two  assumptions  were  easily  satisfied  because 
each  factor  x was  either  contained  in  F or  not  contained  in 
F;  thus,  the  outcomes  were  dichotomized  and  mutually  exclu- 
sive. The  assumption  of  independence  between  trials  was  not 
strictly  satisfied.  The  assumption  that  each  factor  mention 
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V7as  an  independent  trial  was  based  on  the  presumed  lack  of 
correlation  between  factors.  Thus,  the  existence  of  a posi- 
tive correlation  between  two  factors  would  imply  that  the 
mention  of  one  factor  could  increase  the  likelihood  of  the 
second  factor  being  mentioned.  If  both  factors  were 
elements  of  F,  the  mention  of  one  factor  could  increase  the 
probability  of  the  second  factor  being  mentioned,  and  subse- 
quently, being  an  element  of  F as  well. 

Casual  examination  of  the  factors  included  in  F sug- 
gested that  several  factor  pairs  were  probably  correlated. 
Some  examples  included  Reliability  and  Confidence  in  systems 
or  Understanding  of  systems  and  Degree  of  training.  The 
correlation  between  factors  could  change  the  probability  of 
a factor  being  mentioned;  yet,  no  claim  was  made  that  each 
factor  in  U was  equally  likely  to  be  mentioned.  Rather,  the 
concern  centered  on  the  probability  of  a factor  mentioned 
being  an  element  of  F or  not.  Therefore,  each  individual 
mention  of  a factor  was  treated  as  an  independent  trial 
which  resulted  in  success  or  failure. 

The  assumption  of  the  random  selection  of  events  was 
also  not  strictly  achieved.  Ideally,  the  random  selection 
of  events  should  have  been  made  from  the  population  of  all 
factor  mentions  of  all  users.  Clearly,  this  was  impossible 
within  the  constraints  of  any  reasonable  research.  Instead, 
an  attempt  was  made  to  select  a representative  user  group 
under  certain  randomizing  conditions . The  set  of  factors 
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mentioned  by  these  users  was  assumed  to  comprise  a random 
selection  of  events.  The  sample  selection  procedure  and 
randomizing  conditions  appear  in  the  following  section. 

Sample  Selection  Process 

The  limitations  imposed  by  organizational  access  and 
economy  of  resources  precluded  the  selection  of  a truly 
random  sample.  Therefore,  a representative  sample  was 
employed  to  include  a variety  of  functional  users  from 
different  types  of  organizations.  In  order  to  codify  the 
test  data,  the  following  taxonomy  was  used. 

Eight  organizational  categories  were  selected,  and  a 
local  organization  for  each  category  was  identified.  The 
majority  of  the  categories  was  selected  from  those  used  by 
Datamation,  a computer  trade  journal,  for  survey  studies 
(McLaughlin,  1976).  These  were  augmented  by  other  cate- 
gories which  included  significant  user  groups.  The  catego- 
ries of  organizational  types  represented  in  the  sample  were: 

1.  Academic 

2 . Banking 

3.  Hospital 

4.  Manufacturing 

5.  Military 

6.  Retailing 

7 ■ State  Government 

8.  Utility 
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Seven  functional  areas  were  identified  for  the  purpose 
of  user  selections  within  the  organizations.  These  func- 
tional areas  were: 

1.  Accounting 

2.  Administration 

3 . Finance 

4.  Marketing 

5.  Operations 

6.  Personnel 

7 . Planning /Budget 

8.  Other (supplied  by  the  user  subjects) 

No  attempt  was  made  to  rigidly  require  that  a user  fit 

purely  into  a single  functional  area.  The  user  subjects 
were  asked  to  identfy  all  of  the  functional  areas , including 
any  not  specified,  that  were  applicable  to  their  present  job 
position. 

Within  these  bounds  established  for  a representative 
sample,  an  attempt  was  made  to  randomize  the  selection  as 
much  as  possible.  Each  organizational  category  and  each 
functional  area  were  assigned  a single  alphabetic  character 
and  a digit,  respectively.  A pairing  of  the  functional 
areas  to  the  organizational  categories  was  performed  using 
random  digits  selected  arbitrarily  from  a table  of  random 
numbers  (Selby,  1973:630).  Thirty  such  pairings  were  made. 
The  previously  identified  local  organizations  were  then 
contacted  to  supply  the  name  of  a user  subject  matching  the 


functional  area  that  had  been  randomly  assigned. 

The  individual  who  was  contacted  for  each  organization 
was  given  the  liberty  to  provide  names  of  users  who  did  not 
match  the  requested  functional  areas.  Several  contacts  did 
recommend  and  provide  names  of  users  whose  functions  they 
felt  were  more  appropriate  for  their  organization.  Three 
organizational  contacts  provided  more  than  the  number  of 
names  requested.  These  additional  names  were  included  in 
the  sample.  A total  of  33  users  was  identified. 

Therefore,  the  sample  was  randomized  in  two  ways.  The 
first  was  by  the  randomized  pairing  of  the  functional  areas 
and  the  organizational  category.  The  second  was  by  the 
independent  subject  selection  within  the  organization. 
Typically,  each  organization  had  several  users  who  could 
have  been  selected  from  a given  functional  area.  The  person 
contacted  in  conjunction  with  an  organization  selected  the 
subject  from  that  group  independently. 

Each  subject  identified  in  the  above  manner  was 
contacted  via  a letter  and  asked  to  participate  in  the 
research  project.  The  cover  letter  is  shown  in  Appendix 
3.2.  Only  one  of  the  subjects,  B2 , declined  the  request. 
Those  not  declining  were  contacted  by  telephone  to  arrange 
an  appointment  for  an  interview. 


Each  subject  was  asked  to  complete  a brief  background 
form  before  the  interviews  were  conducted.  The  form  also 
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included  a consent  statement  in  which  the  subject  acknowl- 
edged that  participation  was  voluntary  and  that  permission 
to  record  the  interview  was  granted.  All  of  the  subjects 
completed  the  form  and  signed  the  consent  statement.  The 
blank  form  is  shown  in  Appendix  3.3. 

A distribution  of  the  subjects  by  functional  position, 
experience  level,  type  of  output  interactions,  and  level  of 
usage  is  shown  in  Table  3.4.  The  background  form  to  derive 
this  distribution  was  deliberately  short  and  easy  to 
complete.  This  data  was  of  secondary  interest,  and  the 
collection  was  limited  to  prevent  any  detraction  from  the 
interview  process  itself. 

The  distribution  of  the  subject  sample  supported  the 
contention  that  a representative  sample  was  achieved.  In 
addition  to  the  seven  primary  functional  areas,  eight  more 
areas  were  identified  by  subjects  in  the  Other  category. 
Therefore,  a total  of  fifteen  functional  areas  in  eight 
different  organizational  types  were  included.  The  great 
majority  of  the  users  considered  themselves  heavy  users  and 
had  a great  deal  of  experience  (over  six  years) . This  was 
expected  from  the  manner  of  selecting  the  sample.  Most 
organizational  contacts  tended  to  identify  the  major  users 
of  the  EDP  resource  as  the  primary  candidates  for  inclusion 
in  the  sample. 

The  middle  management  restrictions  on  the  user  popula- 


tion tended  to  identify  users  with  more  experience  than 
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Table  3.4 


Sample  Distribution 


1 

Background 

Data 

Category 

Number  of 
Subjects 

Functional  job  description* 

Accounting 

13 

Administration 

8 

t 

Finance 

6 

Marketing 

3 

Operations 

11 

Personnel 

6 

Planning /budget 

8 

ft 

Other** 

8 

Experience  using  EDP  support 

0-3  years 

4 

4-6  years 

4 

over  6 years 

24 

Type  of  output  mode 

Hard  copy 

32 

t 

Terminals 

21 

Microforms 

19 

User  EDP  utilization 

classification 

Heavy 

24 

Medium 

4 

* 

Light 

4 

* Subjects  were  asked  to  check  all  functional  areas 
included  in  their  present  job  position. 

► **  Other  functions  listed  were  (1)  Maintenance,  (2)  Payroll, 

(3)  Patient  Services  Receivables,  (4)  Management/ 
Operation  Analysis,  (5)  Manufacturing,  (6)  Operations 
Research,  (7)  Materials  Management,  (8)  Product 
Distribution. 

» 


» 
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perhaps  a completely  random  sample  of  all  users  would  have 
achieved.  The  preponderance  of  heavy,  experienced  users, 
however,  increased  the  number  and  types  of  critical 
incidents  that  could  be  discussed  during  the  interview 
process.  There  was  ample  representation  of  the  three 
primary  modes  of  output  available  to  the  user  subjects. 

Therefore,  the  sample  of  users  was  considered  both 
representative  of  and  randomly  selected  from  the  overall 
user  population  of  interest.  This  conclusion  was  based  on 
the  cross-section  of  organizational  types  and  functional  job 
positions  included  in  the  user  sample.  The  selection  of  the 
specific  users  was  performed  both  independently  and  under 
randomizing  conditions. 

Interview  Procedure 

The  purpose  of  the  interviews  with  the  user  subjects 
was  to  elicit  factors  which  they  deemed  important  as  influ- 
encing their  state  of  satisfaction  or  dissatisfaction. 

Three  basic  methods  could  have  been  utilized  to  draw  out 
these  factors  (Herzberg,  Mausner,  and  Synderman,  1959). 

The  first  method  was  the  presentation  of  a list  of 
factors  previously  determined  by  the  investigator.  This 
procedure  was  subject  to  several  distorting  forces.  The 
subject  could  try  to  please  the  investigator  by  rating 
factors  highly  that  were  perceived  to  be  of  high  social 
acceptance  rather  than  factors  which  were  personnally 


important.  The  subject  could  also  react  with  bias  toward  a 
listing  of  factors  assembled  by  "experts"  and  fail  to 
respond  with  pertinent  factors  which  were  not  included  on 
the  list. 

A second  method  was  to  simply  ask  the  subject  to  list 
and  rank  factors  which  were  personally  important.  This 
direct  approach  was  subject  to  the  same  halo  effect  induce- 
ment of  the  first  method.  Moreover,  the  subject  could 
simply  forget  pertinent  factors  without  a proper  stimulus. 

A third  method  which  utilized  critical  incidents  was 

adopted  for  this  interviewing  procedure.  The  critical 

incident  technique  was  used  with  great  success  by  Herzberg, 

Mausner,  and  Synderman  (1959)  in  developing  the  two-factor 

theory  of  job  satisfaction.  Content  analysis  of  critical 

incident  interviews  also  played  an  integral  role  in  the 

development  of  the  Job  Descriptive  Index,  a measurement 

instrument  for  job  satisfaction  (Smith,  Kendall,  and  Hulin, 

1969) . In  describing  the  development  of  the  critical 

incident  technique,  Flanagan  stated: 

It  should  be  emphasized  that  the  critical  incident 
technique  does  not  consist  of  a single  rigid  set  of 
rules  governing  data  collection.  Rather  it  should  be 
thought  of  as  a flexible  set  of  principles  which  must  be 
modified  and  adapted  to  meet  the  specific  situation  at 
hand  (Flanagan,  1954:335). 

The  critical  incidents  for  this  research  were  those 
situations  in  which  a subject  felt  particularly  satisfied  or 
dissatisfied  with  the  computer-based  information  products 


and  services  that  were  received  at  any  time  during  the 
user's  working  career.  By  asking  the  subject  to  describe 
such  incidents  and  by  using  neutral  probes,  a more  reliable 
set  of  factors  was  obtained.  It  was  observed  that  the 
distortions  affecting  the  first  two  methods  of  eliciting 
factors  were  "markedly  less  operative  when  a respondent  was 
talking  about  actual  events"  (Herzberg,  Mausner,  and 
Synderman,  1959:15).  It  was  assumed  that  any  factor 
mentioned  by  a user  while  discussing  a critical  incident  was 
important  to  that  user. 

The  descriptions  of  the  critical  incidents  by  the 
subjects  were  probed  in  a semi- structured  manner.  Since  the 
subject  was  given  freedom  to  select  his  events,  the  inves- 
tigator was  forced  to  probe  in  a manner  that  was  consistent 
with  these  descriptions  in  order  to  secure  the  pertinent 
information.  A complete  interview  guideline  with  suggested 
probes  is  included  in  Apppendix  3.4. 

The  interview  procedure  was  pre-tested  to  determine  the 
effectiveness  of  the  probes  and  the  clarity  of  the  instruc- 
tions. As  a prelude  to  the  actual  interviewing  process,  the 
Interviewer ' s Manual  published  by  the  Survey  Research  Center 
at  the  University  of  Michigan  (1969)  was  extensively  stud- 
ied. Several  other  references  on  interviewing  techniques 
were  consulted  to  alleviate  any  lack  of  interviewing 
experience  (Adams,  1958;  Cannel  and  Kahn,  1968;  Converse  and 
Schuman,  1974;  Dexter,  1964;  Hyman,  1954;  Yates,  1975). 


Four  test  subjects  (not  included  in  the  sample  group) 
were  interviewed  using  the  guidelines  and  suggested  probes. 
Following  the  interview,  each  subject  was  asked  to  evaluate 
the  interviewing  process.  All  of  the  test  subjects  express- 
ed extremely  favorable  reactions  to  the  interview  technique. 
Three  of  the  subjects  stated  that  they  "forgot"  that  it  was 
being  taped  during  the  interview.  The  unstructured  format 
and  the  use  of  critical  incidents  seemed  especially  effec- 
tive in  stimulating  the  subjects'  recall. 

One  test  subject  was  asked  to  estimate  the  number  of 
factors  that  he  had  mentioned  during  his  interview.  His 
ee-timate  was  12  factors;  yet,  a joint  review  of  the  inter- 
view recording  revealed  31  factors  mentioned  by  the  subject. 
During  the  replay  of  the  interview,  the  subject  expressed 
surprise  at  many  of  the  factors  that  he  had  mentioned  in 
describing  certain  incidents.  This  exchange  strengthened 
the  confidence  that  the  critical  incident  technique  was 
appropriate  for  this  data  collection  purpose. 

Data  Extractions 

The  interviews  were  recorded  on  cassette  tape  after 
obtaining  permission  from  the  subject.  No  subject  declined 
to  have  the  interview  recorded.  An  analysis  of  the  recorded 
interviews  was  used  to  extract  the  "thought  units"  from  the 
interview  statements.  Herzberg,  Mausner,  and  Synderman 
(1959:33)  defined  a thought  unit  as  "a  statement  about  a 
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single  event  or  condition  that  led  to  a feeling,  a single 
characterization  of  a feeling,  or  a description  of  a single 
effect."  Some  examples  of  thought  units  selected  at  random 
from  the  interview  recordings  are: 

1.  "If  we  have  problems  in  getting  those  problems 
solved,  or  getting  a solution  or  change  to  a program,  an 
alteration  of  any  type,  it  is  like  an  act  of  Congress  to  get 
it  done." 

2.  "We  just  do  not  have  the  control  that  we  had 
before. " 

3.  "The  only  thing  you  run  into,  as  far  as  a problem, 
is  the  currency  of  information  that  is  on  the  file." 

4.  "I  think  that  what  has  been  lacking  is  user 
participation. " 

5.  "I  think  that  we  have  created  a lot  of  programs 
that  have  yielded  very  little  useful  data." 

6.  "There  is  a lack  of  understanding  of  the  products 
that  we  receive,  and  really  the  overall  system,  within  the 
users,  particularly  the  mid- level  managers." 

Each  of  these  thought  units  expressed  a factor  which 
influenced  the  subject's  satisfaction.  The  exact  procedures 
for  the  factor  extraction  process  were  included  in  a 
syllabus  developed  to  standardize  the  methodology.  The 
syllabus  is  shown  in  Appendix  3.5.  The  procedural  steps  of 
the  syllabus  are  listed  in  Figure  3.2. 
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1.  Listen  to  the  entire  tape  and  concentrate  on  recognizing 
thought  units  as  they  are  expessed  by  the  respondent. 

2.  Rewind  the  tape. 

3.  Again,  listen  to  the  tape,  but  stop  anytime  that  a 
thought  unit  is  expressed. 

4.  If  the  factor  expressed  by  the  thought  unit  is  clear, 
write  down  the  factor  directly.  If  the  factor  is  not  clear, 
write  down  the  thought  unit  verbatim. 

5.  Evaluate  the  thought  unit  before  proceeding  with  the 
tape.  However,  a short  portion  of  the  interview  beyond  the 
thought  unit  in  question  may  clarify  it.  Decide  on  a factor 
that  the  thought  unit  expresses  and  write  it  down. 

6.  Proceed  with  the  tape  and  repeat  steps  3-5  until  the 
interview  portion  is  completed. 

7.  Divide  the  list  of  factors  that  were  extracted  from  the 
tape  into  two  groups.  Those  factors  that  were  defined  in 
Part  III  of  this  syllabus  are  listed  in  the  first  group 
using  the  nomenclature  as  defined.  The  second  group  of 
factors  contain  those  factors  not  defined  in  this  syllabus. 
The  nomenclature  of  these  factors  must  be  devised  to  convey 
understanding  about  the  undefined  factors.  Duplicate 
factors  are  listed  only  once  in  the  final  two  groups. 

8.  This  completes  the  data  extraction. 


lb.  k 
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Figure  3 . 2 

Syllabus  Procedures  for  Interview  Evaluations 
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One  weakness  in  the  procedure  was  that  the  investigator 
performed  all  of  the  interviews  and  extracted  all  of  the 
factors  for  each  subject.  There  were  no  factors  inten- 
tionally mislabeled,  however,  to  conform  to  the  a priori 
list  of  factors  proposed  by  the  investigator.  The  potential 
for  judgmental  bias  did  exist,  and  a simple  test  to  evaluate 
the  strength  of  that  bias  was  performed. 

A test  subject's  recording  was  selected  for  the 
evaluation.  A test  subject  was  used  instead  of  a sample 
subject  to  protect  the  confidentiality  that  had  been 
promised  to  each  sample  subject.  The  test  subject's 
permission  was  obtained  prior  to  the  use  of  the  interview 
recording  for  this  purpose. 

Four  judges  were  selected,  and  each  judge  was  given  a 
copy  of  the  syllabus  for  data  extraction  and  instructed  on 
its  use.  The  interview  recording  and  cassette  player  was 
provided  to  each  judge,  in  turn,  to  perform  a data  extrac- 
tion. In  addition,  the  investigator  performed  a data 
extraction  from  the  recorded  interview.  There  was  no 
communication  between  the  judges  about  the  results  which 
they  obtained  until  all  had  been  collected  by  the 
investigator. 

A comparison  of  the  results  is  shown  in  Table  3.5. 
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Each  factor  identified  by  either  the  investigator  or  one  of 
the  independent  judges  is  listed  in  the  factor  set.  The 
factors  extracted  by  the  investigator  are  signified  by  an 
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Table  3.5 

Data  Extraction  Comparison 


Factor 

Judges 

Composite 

1 

T 

r 

T-5 

Priorities  determination 

X 

X 

X 

X 

X 

X 

X 

Organizational  position  of  EDP  function 

X 

X 

X 

X 

X 

Relationship  with  the  EDP  staff 

X 

X 

X 

X 

X 

X 

X 

Communication  with  the  EDP  staff 

X 

X 

X 

X 

X 

X 

X 

Technical  competence  of  the  EDP  staff 

X 

X 

X 

X 

X 

X 

Attitude  of  the  EDP  staff 

X 

X 

X 

X 

X 

X 

Schedule  of  products  and  services 

X 

X 

X 

X 

X 

X 

Time  required  for  new  development 

X 

X 

X 

X 

X 

X 

Processing  of  change  requests 

X 

X 

X 

X 

X 

X 

X 

Response/ turnaround  time 

X 

X 

X 

X 

X 

X 

X 

Mode  of  interface 

X 

X 

X 

X 

X 

X 

Convenience  of  access 

X 

X 

X 

X 

X 

X 

Accuracy 

X 

X 

X 

X 

X 

X 

Timeliness 

X 

X 

X 

X 

X 

Precision 

X 

X 

Reliability 

X 

X 

X 

X 

X 

X 

Currency 

X 

X 

X 

X 

X 

Completeness 

X 

X 

X 

X 

X 

X 

Flexibility 

X 

X 

X 

X 

X 

X 

X 

Format  of  output 

X 

X 

X 

X 

Language 

X 

X 

X 

X 

X 

Volume  of  output 

X 

X 

X 

X 

X 

Relevancy 

X 

X 

X 

X 

X 

Error  Recovery 

X 

X 

X 

X 

X 

X 

Documentation 

X 

X 

X 

X 

Expectations 

X 

X 

X 

X 

X 

Understanding  of  systems 

X 

X 

X 

X 

X 

X 

Perceived  utility 

X 

X 

X 

Confidence  in  systems 

X 

X 

X 

X 

X 

X 

X 

Feeling  of  participation 

X 

X 

X 

X 

X 

X 

X 

Feeling  of  control 

X 

X 

X 

X 

V 

iV 

X 

X 

Degree  of  training 

X 

X 

X 

X 

Job  effects 

X 

X 

X 

X 

X 

X 

X 

Continuity  of  EDP  staff* 

X 

X 

X 

X 

X 

Long  term  trend  of  improvement* 

X 

X 

Increased  feeling  of  professionalism*,** 

X 

X 

* Factor  not  included  in  F 

**  Judge  3 was  undecided  if  this  was  a separate  factor  or 
one  aspect  of  the  Job  effect  factor. 
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"X"  in  column  one.  The  factors  extracted  by  the  independent 
judges  are  similarly  identified  in  columns  two  through  five. 
A composite  comparison  contrasts  the  factors  identified  by 
the  investigator  with  the  pooled  set  of  factors  identified 
by  the  independent  judges. 

The  results  illustrated  in  Table  3.5  indicated  that 
independent  judges  may  reach  some  different  conclusions 
about  the  factors  discussed  by  a subject.  However,  there 
was  a strong  tendency  of  judges  to  agree  on  most  factors. 
There  were  36  factors  mentioned  by  at  least  one  judge.  The 
mean  number  of  judges  extracting  the  same  factor  was  3.6 
(maximum  was  5.0).  A complete  comparison  between  each  pair 
of  judges  on  the  factor  set  indicated  that  perfect  agreement 
between  each  pair  of  judges  would  have  resulted  in  360 
matches.  There  were  232  matches  achieved.  Based  on  these 
frequencies  for  this  sample,  the  percentage  of  agreement  was 
64.4  percent.  If  all  five  judges  had  identified  each  of  the 
36  factors,  there  would  have  been  180  identifications. 

There  were  129  identifications  for  a rate  of  71.7  percent. 
This  indicated  that  an  average  of  71.7  percent  of  all  judges 
identified  the  individual  factors  of  the  factor  set. 

A comparison  of  the  factor  set  identified  by  the  inves- 
tigator with  the  composite  set  of  factors  identified  by  the 
four  independent  judges  revealed  32  matches  and  only  4 mis- 
matches. The  investigator  did  not  extract  a single  factor 
that  was  not  identified  by  at  least  one  of  the  independent 


judges.  The  composite  comparison  is  also  illustrated  in 
Table  3.5. 

This  independent  extraction  comparison  supported  the 
assumption  that,  although  there  would  be  some  inter- judge 
disagreement  on  the  factors  contained  in  an  interview 
recording,  the  factors  identified  by  the  investigator  would 
not  differ  significantly  from  those  identified  by  inde- 
pendent judges  utilizing  the  data  extraction  syllabus.  The 
extensive  amount  of  time  required  of  the  judges  to  perform 
the  data  extractions  precluded  a more  comprehensive  test  of 
this  assumption. 

Test  Results 

The  data  to  test  the  hypotheses  were  collected  after 
the  interview  and  data  extraction  procedures  were  developed 
and  evaluated.  Thirty- two  sample  subjects  were  interviewed. 
The  factors  were  extracted  from  each  interview  in  accordance 
with  the  procedures  outlined  in  the  syllabus.  Each  subject 
was  provided  with  a list  of  the  factors  extracted  only  from 
his  interview.  Operational  definitions  of  each  factor  were 
also  provided.  The  subjects  were  asked  to  rank  each  factor 
in  order  of  importance  as  an  influence  on  their  state  of 
satisfaction  or  dissatisfaction.  Only  one  subject,  B3 , 
failed  to  rank  the  factors.  The  factors  mentioned  by  each 
subject  and  the  subject  rankings  of  those  factors  in  order 
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of  importance  are  shown  in  Appendices  3.6  and  3.7,  respec- 
tively. 

The  total  number  of  factors  mentioned  by  the  subjects 
was  638.  The  number  of  factor  mentions  per  subject  ranged 
from  15  to  31  with  an  average  of  19.9  factors  per  subject. 
Only  13  of  the  638  factor  mentions  were  not  included  in  F. 
Therefore,  625  of  the  factor  mentions  were  designated  as 
"successes"  and  13  were  designated  as  "failures".  Due  to 
the  large  number  of  trials,  n = 638,  a normal  approximation 
of  the  binomial  distribution,  corrected  for  continuity,  was 
used  to  calculate  the  test  statistic.  A restatement  of  the 
hypotheses  is  presented  below: 

H0  : Pr  (x  e F)  < .90 
: Pr  (x  e F)  > .90 

The  "critical"  or  rejection  region  was  defined  as  any  z 
statistic  greater  than  2.327  for  a level  of  significance  of 
.01.  The  z statistic  for  625  successes  out  of  638  trials, 
corrected  for  continuity,  was  6.768.  Since  this  value 
exceeded  2.327,  the  null  hypothesis  was  rejected,  and  the 
alternative  hypothesis 

Hj  : Pr  (x  e F)  > .90 

was  accepted.  Therefore,  it  was  concluded  that  the  domain 
of  the  user  satisfaction  construct  was  defined  by  the 
factors  contained  in  the  subset  F.  This  conclusion  was 
supported  at  the  .01  level  of  significance. 
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The  rankings  of  the  factors  by  the  user  subjects  were 
also  revealing.  The  frequency  of  mentions,  minimum  rank, 
and  maximum  rank  for  each  factor  are  shown  in  Table  3.6. 

The  "average"  rank  for  each  factor  is  also  shown.  Care  must 
be  exercised  in  any  interpretation  based  upon  this  "average" 
because  it  is  a mechanical  average  based  strictly  on  ordinal 
data.  True  "means"  require  interval  data.  The  average  rank 
was  calculated  only  to  give  a "feel"  for  the  ranking  distri- 
bution for  each  factor. 

The  user  rankings  dramatically  underscore  the  assump- 
tion that  different  users  differ  widely  in  their  perception 
of  the  importance  of  different  factors.  Eighteen  of  the  38 
factors  contained  in  F were  ranked  first  in  importance  by  at 
least  one  subject.  Only  3 of  the  38  factors  had  a minimum 
rank  greater  than  10.  Yet,  the  maximum  ranks  for  each 
factor  was  not  less  than  12,  with  24  of  the  factors  having 
maximum  ranks  greater  than  20.  These  wide  ranges  of  ranks 
assigned  to  the  factors  supported  the  assumption  that  users 
differentiate  substantially  in  the  order  of  importance  of 
the  individual  factors . 

The  factor  rankings  of  the  factors  not  contained  in  F 
singled  out  one  factor  for  additional  consideration.  Inte- 
gration of  systems  was  mentioned  by  four  subjects  and  ranked 
2,  2,  7,  and  17  in  importance.  The  significant  combination 
of  frequency  of  mentions  and  high  overall  rankings  provided 
evidence  that  F should  be  expanded  to  include  this  factor. 


T - - ■ ■ 


LPW,r' 


v 3W1 


. 


L ^ 


'sMnoocNjmcncvir^ooocooorWOHi-tcoovtmoN 

CNONONOO<nr4r4000Nf>Oi-IO\aMna\vOvOOOvDOOvO 


cMOcvjo\'d-<T><Nr^mo\<ricMoooor-)cr>v£>r^cMooooro 

C^nNHC'JNNOJNHHCSCNIHCSIHHHHHHN 


i— I 1-4  04  ■<J‘  *4“  r4  i— I r4  C\|  ON  t-4  U3  <1*  HOOHHHHHHH 


in>d-(n(ooMfONi-i(^ONvoMnNtoin<finH(n 

<N  i — I C4  CNI  04  C4  , — I 04  04  HNHNH  H i — I CM 


44 

•H 

d 

d 

to  o 

P 

<0  t4 

O 44 

o 

•rl  O 

o 

> d 

w 

Vi  P <44 

(U  <44  <14 

(U 

<0  (0 

43 

O 44 

44 

Vi  Q to 

to 

44 

O W <44 

<U 

d 

43 

<44  <44  <44  O 

O 

(U 

44 

<U<44  tO  Q 

t4 

a 

t4 

44  42  tO  44  td 

> 

O to 

s 

d 44  44  to 

Vt 

0 44 

tu  to  0» 

tu  1-4  tO 

44  d 

a <44  0 43 

CO 

tu  (U 

c o 

O O O 44  <44 

> p 

<U  -rl  d <3  QW  <44  <U  CT 
B40P.CN  <44  (0  d T)  <U 
(U  t4  t4  o 01  O 4 d V4 
> 44  44  <44  m-I  (U  43  CO  & 
r-4<U(0O44rd44(U  <0  <U  <U 


> 44  44  44  t4  (U  43 

r-l  <U  ttj  O 44  43  44 

O O.C  v4  44 

> 6 t4  T3  to  43 

C O S O O 43  -U 

•rl  O H 43  (Hi  4-1  t4 

<U  44  -H  S 
44  ,-t  44  (Ur-4  S 

d <0  tu  0 to  d 

tu  c T3  C P.O 

So  AS  O -rl  -I-I 
tu  *rl  to  O -r4  4C  44 
6044  <U  tO  44  to  <0 
to  (U  t4  43  ctJ  c O 
C N 44  | N O'H 
tO  -H  v4  0)  *H  *rl  d 

B 13  U M d 44  3 
(0  O Vi  (0  <0  S 

Q,  60 'H  (0  60r-l  g 

O Vl  UJ2  K 0)  O 
HOMJONU 


0 CV  44  d W) 

J3Q  O d 

0)  W 3>  U (0 
44  -o  043 

01  V 004  u 
0.43  Vl  4 

0 44  CUT)  14-1  I 
O 0)  O I 

o *4-1  <4-4  Vt  | 
O O -rl  60  I 

r-l  p d ; 

to  <u  <u  cr-H  ' 

OTfrl  01  to 
•H  3 3 h to  ! 

d 44  t3  <U  ' 
43  f-l  <U  <U  O 
U44J3  B o 
(U  44  O H Vi 
H <3  CO  H P*  C 


P tu 
o o 
Vi  to 

44  (0  <44  < 


CV  P 44  (U  >n  to 

0,44  dO  to  44  (U 

P -H  d to  d "rl  d 

to  <U  (U  >iOI  OH  Ml 

to  <44  t4  O d t4  "rl  O 44 

V d O d <0-4  tOJ3  c 01 

OO  (UVlr4i4tOtUr4 
T30<U>P<UO-HViO 
d tofl  d U B Olrl  It  B 
<U(UOOOi4Vl<UPO 
>dSU<HMt!UU 


J 


ooiHc-jvocMOOooomr^o'ioor^cnoo 

• ••••••••••••••a 

iTKfN'ONOMnOiriHO'OOCMNN 

r-H  i — I I — i l — I i — I 1 — 1 i — I i — I I — I i — I i — t 1 — I 


•jco^mrvrvOH>iTg<l-vovor)CMN 

r— icscsicSi-Hi— icmcncmc'JonIi-ic'sIcncnxn 


cNicnovOi— irooo<fr^r-icsii— Ii-hcnloi-h 


r-~mr^rHcN<T>vocncNir-i'J-i— ir^ocsioo 

H H i — I N HNOJNrHHCNHCVI 


V4  4J 

4-4 

cd 

4-1  4-* 

O G 

G 

4-4 

O -H 

4J  qu 

Pu 

>N  cd 

r— 1 

O 4-4 

4-1 

J-l  43 

G 00-H 

cd  G 

G 

a) 

O CO  G 4-1 

Ph  >*.  O 

O 

> 4H 

•H  d -H  G 

4-4 

O O 

4-4  O 

iH  4-4 

4H 

>*.  o 

Cd  t4  g 'O 

i-H  O 0)  O O <0  >-.4-4 
•H  00  C M 4-4  C 

x>  4-4  cd  0)  03  v4  0) 

x E oo3  ai  o 3 3 

J)  fi  (JHH  ^ y u 
HOHOOIWOJO 
tn  ti,  ^ > pi  w w q 


g 

o 

•H 

■U 

cd 

(X 

•H  r— I 00 
O O G 

>-H  }-4  "H 
■U  4J  C 
V4  Ci-rH 

cd  o cd 

ftO  M 

4 -4  CO 
44  44  4J 
0 0 4-10 
O <U 
00  00  4-1 

G C a)4-t 


u co  qj  g o a) 
Cd  4-4  > 0)  00  00  44 
4-1  CO  1-IT3  C C 0)4-4 
O V4  0)  -H  -H  -H  0)  0) 
0)  a)  O 4-4  rH  r-l  *4 

O-TJ  M C 0)  <u  oox> 

XG<HO<U<U<UO 

W5P.UhhOh! 


1 


124 

Although  this  expansion  was  unnecessary  to  satisfy  the 

statistical  criteria  that  had  been  established,  the  judgment 

that  F was  improved  by  including  this  factor  prevailed.  The 

operational  definition  was  formulated  as  follows: 

Integration  of  systems:  The  ability  of  systems  to 
communicate/ transmit  data  between  systems  servicing 
different  functional  areas. 

This  expansion  extended  the  number  of  factors  in  F to 
39.  The  overall  conclusions  are  presented  in  the  following 
section. 

SUMMARY 

The  objective  of  this  research  phase  was  the  identi- 
fication of  a subset  of  factors  that  form  the  content  domain 
of  the  user  satisfaction  construct.  Thirty-eight  factors 
were  identified  via  a literature  review  and  an  independent 
assessment  by  a group  of  expert  judges.  A representative 
sample  of  32  users  was  interviewed  to  elicit  factors  impor- 
tant to  their  satisfaction  state.  The  data  extracted  from 
the  interviews  formed  the  basis  of  an  empirical  test  of  the 
completeness  of  the  factor  list.  The  test  results  estab- 
lished that  the  domain  of  the  user  satisfaction  construct 
was  defined  by  the  factors  contained  on  the  list  at  the  .01 
level  of  significance.  One  additional  factor  was  added  to 
the  domain  because  of  the  high  importance  ranking?  it 
achieved.  The  probability  exceeded  .90  that  a factor 
considered  important  by  a user  was  contained  in  the  domain 


as  defined. 

These  39  factors  are  listed  in  Appendix  3.8.  Each 
factor  was  designated  as  F^,  i = 1 to  39,  for  the  remainder 
of  this  study.  The  factor  listing  is  provided  on  a fold-out 
sheet  for  the  reader's  convenience.  The  definitions  are 
also  given. 

The  results  of  this  chapter  established  the  domain  of 
the  user  satisfaction  construct  and  provided  the  fundamental 
basis  to  answer  the  question  of  "what"  to  measure  to  indi- 
cate the  state  of  an  individual  user's  satisfaction.  The 
next  chapter  traces  the  development  of  the  measurement 
instrument  which  combines  the  domain  definition  with  the 
semantic  differential  methodology. 
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CONSTRUCTION  OF  THE  MEASUREMENT  INSTRUMENT 

The  construction  of  the  measurement  instrument 
integrates  the  domain  of  the  user  satisfaction  construct 
with  the  semantic  differential  methodology.  The  domain 
consists  of  those  factors  identified  in  Chapter  3 which 
influence  user  satisfaction.  The  semantic  differential 
methodology  presented  in  Chapter  2 requires  that  scales, 
defined  by  pairs  of  bipolar  adjectives,  be  constructed  for 
each  factor.  The  selection  of  appropriate  adjective  pairs 
to  anchor  opposite  poles  of  the  evaluation  continuum  for 
each  factor  is  critical  to  the  measurement  process. 

The  selection  procedure  to  identify  bipolar  adjective 
pairs  for  the  specific  scales  is  described  in  this  chapter. 
The  procedure,  conducted  in  concert  with  five  university 
professors,  results  in  the  formulation  of  four  evaluation 
scales  for  each  user  satisfaction  factor.  These  four  scales 
are  augmented  by  an  internal  consistency  scale  and  an 
importance  weighting  scale  to  operationalize  the  measurement 
of  a user's  feeling  toward  each  factor. 

The  second  portion  of  this  chapter  centers  on  the 
decisions  of  format  that  influence  the  effectiveness  of  the 
measurement  instrument.  These  decisions  lead  to  the  initial 
assembly  of  the  instrument  in  booklet  form  with  appropriate 
instructions  for  completion.  A preliminary  evaluation  of 
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the  instrument  by  a sample  group  of  users  establishes  that 
the  instrument  and  instructions  have  the  desired  charac- 
teristics of  clarity,  brevity,  and  visual  appeal. 

The  completed  measurement  instrument  described  in  this 
chapter  provides  a means  to  measure  user  satisfaction.  A 
complete  copy  of  the  instrument  is  presented  in  Appendix 
4.1. 


SELECTION  OF  SCALES 

The  scales  of  a semantic  differential  measurement 
instrument  are  defined  by  bipolar  adjective  pairs  which 
describe  the  concept  or  factor  that  is  to  be  measured.  The 
initial  step  in  the  construction  of  a user  satisfaction 
measurement  instrument  was  the  selection  of  adjective  pairs 
for  each  of  the  39  factors  identified  in  Chapter  3 as  the 
construct  domain. 


The  selection  procedures  employed  in  other  studies 
typically  utilized  only  adjective  pairs  from  the  original- 
thesaurus  study  by  Osgood.  Coyne  and  Holzman  (1966:665) 
commented,  "Too  often  investigators  cull  items  from  the 
Osgood,  Suci,  and  Tannenbaum  (1957)  book  without  pretesting 
their  relevance  or  their  factor  structure  for  the  concept 


they  wish  to  measure." 


described  by  Lusk: 


The  effect  of  this  practice  was 
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By  selecting  adjectives  based  solely  upon  Osgood's 
thesaurus  study,  a high  preponderance  of  mid- interval 
(neutral)  responses  may  be  evidenced  indicating  an 
inapplicable  concept/adjective  pairing  (Lusk,  1973:202). 

Lusk,  however,  followed  this  observation  with  a 
recommended  selection  procedure  that  depended  directly  on 
the  use  of  adjective  pairs  from  Osgood's  study.  The  proce- 
dure utilized  a sample  of  individuals  representative  of  the 
subject  population  of  interest.  The  semantic  responses  of 
the  sample  group  were  used  to  calculate  the  variance  from 
the  midpoint  for  each  adjective  pair  (selected  from  Osgood's 
study)  for  the  measurement  concepts.  The  adjective  pairs 
with  the  largest  variance  were  then  selected  to  form  the 
measurement  instrument. 

A factor  analysis  procedure  was  employed  by  Summey 
(1974)  to  select  adjective  pairs.  He  first  selected 
adjective  pairs  which  were  identified  by  interview  descrip- 
tions of  the  concepts  to  be  measured.  This  set  of  adjective 
pairs  was  then  administered  to  a pretest  subject  group.  The 
responses  were  subjected  to  a factor  analysis,  and  the 
adjective  pairs  receiving  the  highest  loadings  on  the 
desired  semantic  space  dimensions  were  selected  for  the 
me  a s ur  emen  t ins  t rumen  t . 

The  adjective  pairs  for  the  user  satisfaction  instru- 
ment were  selected  to  satisfy  three  criteria: 

1.  Factorial  composition 

2.  Relevance 
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3.  Semantic  stability 

The  factorial  composition  was  intended  to  be  uni- 
dimensional with  all  adjective  pairs  denoting  the  Evaluation 
dimension.  Each  adjective  pair  was  to  be  relevant  to  the 
specific  factor  that  it  measured.  Finally,  semantic 
stability  required  an  unambiguous  polarity  for  each  adjec- 
tive pair.  The  selection  procedure  described  in  the 
following  section  was  designed  to  satisfy  the  specified 
criteria  and  to  minimize  any  bias  or  lack  of  perception 
which  might  exist  in  the  selections  of  a single 
investigator. 

Selection  Procedure 

A two-step,  Delphi  type  procedure  was  used  to  select 
the  adjective  pairs  for  the  user  satisfaction  instrument. 
Five  university  professors  and  the  investigator  participated 
in  this  phase  of  the  research.  The  rationale  for  the  use  of 
these  individuals  was  two-fold.  First,  each  had  a back- 
ground of  involvement  in  EDP  activiites.  Secondly,  it  was 
felt  that  the  academic  experience  suggested  a language 
fluency  that  would  be  beneficial  in  the  selection  of 
appropriate  adjectives. 

Preparation  for  the  selection  procedure  involved  two 
preliminary  actions.  The  first  was  the  formulation  of 
stimulus  phrases  that  were  indicative  of  the  individual 
factor  definitions.  Since  the  factor  name  did  not,  in  all 
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A rule  of  thumb  is  that  more  than  10  bipolar  adjectives 
per  concept  may  distract  the  respondent  introducing  a 
degree  of  arbitrariness  in  the  subject's  response. 
Therefore,  if  all  three  factor  dimensions  (Evaluative, 
Potency,  and  Activity)  are  to  be  measured,  a mix  of 
adjectives  in  the  ratio  4 : 3 : 3 is  not  uncommon  (Lusk, 
1973:203) . 

In  addition  to  historical  convention,  Keise  (1970) 
found  that  adequate  sensitivity  (ability  to  discriminate 
between  individuals)  was  achieved  with  four  scales  per 
dimension.  The  results  of  a study  by  Norman  (1959)  indi- 
cated that  the  greatest  reliability  gain  over  single  scale 
ratings  was  attained  by  averaging  three  or  four  scales.  He 
also  noted  that  stability  did  not  significantly  increase 
when  more  than  four  scales  (up  to  eight  scales  in  his  study) 
were  used  for  a single  concept.  The  decision  to  use  four 
adjective  pairs  per  factor  was  made  cognizant  of  these 
results . 
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The  choice  of  four  scales  per  factor  was  also  influ- 
enced by  the  time  required  of  the  subjects . Osgood,  Suci, 
and  Tannenbaum  (1957)  and  Heise  (1970)  observed  that  sub- 
jects could  make  approximately  400+  ratings  per  hour.  It 
was  felt  that  an  upper  bound  of  subject  time  to  complete  the 
instrument  should  not  exceed  30  minutes  (or  200  scale 
ratings).  This  judgment  was  based  upon  subject  behavior 
that  was  observed  during  the  interview  process  and  cautions 
expressed  in  the  literature  (Babbie,  1975) . The  use  of  four 
scales  for  39  factors  required  156  ratings  by  the  subject 
which  was  within  the  upper  bound. 

The  first  step  of  the  selection  procedure  required  each 
judge  to  independently  propose  four  adjective  pairs  for  each 
factor  that  could  be  used  to  evaluate  the  individual  fac- 
tors. Each  judge  was  provided  a list  of  the  39  user  satis- 
faction factors  (with  extended  phraseology)  and  instructions 
for  adjective  pair  specification.  The  complete  set  of 
instructions  is  contained  in  Appendix  4.2. 

The  judges  were  also  provided  a sample  list  of  50 
adjective  pairs  that  had  been  used  in  previous  studies 
utilizing  the  semantic  differential  (Nunnally,  1961) . These 
were  provided  only  as  a guide.  The  instructions  explicitly 
required  that  adjective  pairs  be  selected  that  were  highly 
germane  to  each  factor  with  a clear  polarity  of  "goodness" 
or  "badness".  Pairs  not  found  in  the  sample  list  were 
encouraged  where  appropriate. 


The  specified  adjective  pairs  from  the  judges  and  the 
investigator  were  grouped  by  factor  and  redistributed  to  the 
judges.  A new  set  of  instructions,  shown  in  Appendix  4.2, 
directed  the  judges  to  select  the  best  four  pairs  from  the 
set  for  each  factor.  The  instructions  also  called  for  each 
judge  to  indicate  the  "positive  pole"  of  each  selected 
adjective  pair.  The  "positive  pole"  identified  the  single 
adjective  of  a pair  that  described  a good  or  satisfactory 
state  for  the  specified  factor. 

The  adjective  pairs  were  rank-ordered  by  the  number  of 
judges  selecting  them.  The  four  pairs  designated  by  the 
greatest  number  of  judges  were  selected  for  the  user 
satisfaction  measurement  instrument.  The  investigator 
decided  all  ties.  Any  adjective  pair  that  did  not  have 
complete  agreement  on  polarity  was  automatically  discarded. 
Disagreement  was  regarded  as  an  indicator  of  semantic 
instability,  a violation  of  the  selection  criteria.  Only 
one  adjective  pair  was  eliminated  for  this  reason.  The 
adjective  pairs  selected  by  this  procedure  are  shown  in 
Appendix  4.3. 


Weighting  Function  Scale 

In  addition  to  the  four  adjective  pairs  selected  to 
evaluate  each  factor,  a scale  defined  by  the  adjective  pair 
» "important  - unimportant"  was  used  to  indicate  the  impor- 

tance of  each  factor  to  the  subject.  The  same  response 


» 


categories  were  used  for  each  scale.  The  use  of  a separate, 
but  similar  format,  scale  for  importance  was  successfully 
employed  by  Friedlander  (1965)  and  Mobley  and  Locke  (1970) . 

Internal  Consistency  Scale 

The  adjective  pairs  were  selected  to  evaluate  a 
subject's  feelings  of  satisfaction  toward  distinguishable 
factors.  Homogeneity  would  be  evidenced  by  high  inter- 
correlations between  the  scale  ratings  measuring  a partic- 
ular factor  (Robinson,  Rush,  and  Head,  1974).  An  additional 
scale  was  included  to  provide  another  measure  of  the  inter- 
nal consistency  of  the  instrument.  The  scale  was  defined  by 
"satisfactory  - unsatisfactory".  This  scale  was  included 
with  the  evaluation  scales  and  the  weight  function  scale  for 
each  factor  to  allow  an  item  by  item  comparison  as  well  as  a 
factor  score  comparison  *7ith  a "direct"  assessment  of 
satisfaction.  The  organization  of  scales  into  a completed 
instrument  is  discussed  in  the  following  section. 

FORMAT  OF  THE  INSTRUMENT 
The  selection  of  the  adjective  pairs  to  form  the 
individual  scales  for  each  of  the  39  user  satisfaction 
factors  finalized  the  content  portion  of  the  measurement 
instrument.  The  physical  arrangement  of  this  content  for 
presentation  to  the  subject  groups  involved  specific 
questions  of  format.  Babbie  noted: 
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The  format  of  a questionnaire  can  be  just  as 
important  as  the  nature  and  wording  of  the  questions 
asked.  An  improperly  laid  out  questionnaire  can  lead 
respondents  to  miss  questions,  can  confuse  them  as  to 
the  nature  of  the  data  desired,  and,  in  the  extreme  can 
lead  to  respondents  throwing  the  questionnaire  away. . . 

As  a general  rule  the  questionnaire  should  be  spread  out 
and  uncluttered.  The  researcher  should  maximize  the 
"white  space"  in  his  instrument  (Babbie,  1975:  111). 

The  effect  of  "spreading  out"  the  content  of  a ques- 
tionnaire is  an  increase  in  the  number  of  pages  required  for 
the  same  content  expressed  in  a compressed  format.  Although 
this  effect  gives  the  general  appearance  of  a longer,  more 
time  demanding  questionnaire  to  the  subjects,  experienced 
researchers  report  that  questionnaires  with  clear,  unclut- 
tered formats  are  more  favorably  received  by  respondents 
than  shorter  (fewer  pages)  questionnaires  with  more 
cluttered  formats  (Oppenheim,  1966)  . 

A study  conducted  by  the  Survey  Research  Center  of 
Arizona  State  University  supported  this  finding  (Axelrod, 
1976).  The  study  utilized  two  formats.  One  employed  a 
compact  matrix  response  space  (a  very  crowded  format) 
contained  on  both  sides  of  a single  page.  The  other  format 
was  the  same  content,  spread  out  with  individual  response 
spaces,  contained  in  a 26  page  booklet.  The  two  ques- 
tionnaires were  distributed  randomly  to  the  subject  group. 
Only  44  percent  of  the  single  page  questionnaires  were 
returned,  while  76  percent  of  the  booklet- type  ques- 
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The  higher  response  rate  was  attributed  to  two  primary 
reasons.  The  first  reason  was  easily  detected.  The  booklet 
format  reduced  subject  confusion  and  allowed  a more  rapid 
completion  rate  even  though  more  pages  were  involved.  The 
physical  act  of  page- turning  apparently  positively  rein- 
forced a subject's  sense  of  progress.  In  contrast,  the 
single  page  format  discouraged  subjects  by  the  lack  of 
periodic  "tangible"  evidence  toward  completion. 

The  other  reason  for  the  higher  completion  rate  was  not 
so  obvious.  Follow-up  interviews  revealed  that  a subject's 
commitment  to  complete  the  questionnaire  varied  in  direct 
proportion  to  the  time  spent  in  constructing  the  question- 
naire, as  evidenced  by  the  physical  appearance.  Therefore, 
a questionnaire  that  appeared  "thrown  together"  was  lightly 
regarded  by  the  subjects;  e.g.,  an  attitude  of  "If  the 
investigator  does  not  care,  why  should  I?".  Although  the 
study  results  were  not  codified,  one  general  principle  did 
emerge.  Subjects  did  tend  to  attribute  importance  to  the 
questionnaire  and  the  research  that  it  represented  on  the 
basis  of  the  physical  appearance  of  the  questionnaire. 

Therefore,  three  fundametal  tenets  guided  the  material 
arrangement  of  the  user  satisfaction  measurement  instrument. 
These  tenets  were: 

1.  The  content  was  to  be  unconstricted  with  ample 
"white  soace". 
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2.  The  responses  required  of  the  subjects  were  to  be 
clear,  simple,  and  rapid. 

3.  The  total  design  was  to  reflect  deliberate  fore- 
thought to  achieve  a "professional"  appearance. 

Within  these  basic  guidelines,  the  following  aspects  of 
the  format  were  specif icially  addressed: 

1.  Number  of  scale  intervals 

2.  Choice  of  adverb  modifiers 

3.  Order  of  the  scales 

4.  Order  of  the  factors 

5.  Instruction  set 

6.  Assembly 


Number  of  Scale  Intervals 

The  instrument  scales  were  formed  by  the  separation  of 
the  adjective  pairs  by  a number  of  equal  intervals  which 
represented  response  categories.  Traditionally,  seven 
intervals  were  used  in  research  employing  the  semantic 
differential  (Heise,  1969).  Miller  (1956)  argued  in  support 
of  this  tradition  that  subjects  appeared  to  be  limited  in 
unidimensional  judgments  to  about  seven  categories. 

Gulliksen  (1958)  challenged  that  viewpoint  and  stated  that 
20  - 30  point  scales  were  feasible  for  semantic  differential 
judgments.  Garner  (1960)  confirmed  that  discrimination  did 
increase  as  the  number  of  rating  categories  increased,  but 
Bass,  Cascio,  and  O'Conner  (1974)  reported  that  an  increased 
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number  of  rating  categories  was  also  accompanied  by  an 
increased  overlap  in  adjacent  point  judgments.  This  overlap 
was  as  high  as  44  percent  for  a 9 point  scale. 

The  evidence  was  contradictory.  The  increased  discrim- 
ination achieved  by  the  use  of  many  response  categories  was 
somewhat  offset  by  the  overlap  of  adjacent  point  judgments. 
Furthermore,  as  Garner  observed: 

The  optimum  number  of  rating  categories,  or  at  least  the 
number  beyond  which  there  will  be  no  further  improvement 
in  discrimination,  is  clearly  a function  of  the  amount 
of  discriminability  inherent  in  the  stimuli  being  rated. 
Thus,  there  can  be  no  single  number  of  rating  categories 
appropriate  to  all  rating  situations.  At  the  limits  of 
discriminability,  this  fact  is  obvious  (Garner, 
1960:350). 

A seven  interval  scale  was  used  for  the  user  satis- 
faction measurement  instrument.  The  rationale  for  this 
decision  had  two  main  points.  The  study  of  the  metric 
properties  of  the  semantic  differential  by  Messick  (1957) 
established  the  interval  property  of  the  seven  point  scale 
(reported  in  Chapter  2) . This  was  a very  desirable  property 
that  allowed  a more  sophisticated  statistical  analysis  of 
the  resulting  scores.  This  property  was  not  established  for 
other  than  seven  point  scales. 

The  second  reason  for  the  seven  intervals  was  the 
communication  sensitivity  that  was  achieved.  Osgood,  Suci, 
and  Tannenbaum  (1957:11)  declared  that  the  measurement 
method  "should  yield  differentiations  commensurate  with  the 
natural  units  of  the  material  being  studied,  i.e.,  should  be 
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able  to  reflect  as  fine  distinctions  in  meaning  as  are 
typically  made  in  communicating".  It  was  felt  that  the 
seven  response  categories,  labeled  with  the  polar  adverbs 
extremely,  quite,  and  slightly  on  both  sides  of  the  neutral 
response  category,  did  form  natural  groups  that  reflected 
typical  communication  patterns  describing  user  satisfaction. 
The  use  of  seven  categories  over  four  evaluation  scales 
allowed  a quantitative  distinction  of  28  values  after 
averaging,  ranging  in  mean  scores  from  -3  to  +3  in  steps  of 
.25.  Therefore,  a score  of  +1.5  could  be  interpreted  as 
halfway  between  "slightly  satisfied"  and  "quite  satisfied". 
It  was  not  felt  that  a finer  discriminatory  scale  would 
achieve  a proportionally  greater  explanatory  or  coiumuni- 
cative  power. 

If  a factor  was  not  applicable,  the  subject  was 
instructed  to  check  the  midpoint  of  all  the  scales.  This 
gave  an  effective  score,  weighted  or  unweighted,  of  zero  for 
that  factor's  contribution  to  the  overall  satisfaction 
measure. 

Choice  of  Adverb  Modifiers 

The  metric  characteristics  of  the  adverb  qualifiers 
extremely,  quite,  and  slightly  were  examined  by  Cliff  (1959) 
and  found  to  combine  multiplicatively  with  adjectives  to 
define  rating  positions  of  almost  equidistant  intensity. 

The  neutral  point  of  intensity  was  assumed  to  be  located  at 
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qualification  was  explained  to  the  respondents  in  the 
instructions.  The  adverbs  were  not  repeated  throughout  the 
instrument.  Wells  and  Smith  (1960)  found  that  the  ratings 
achieved  with  and  without  the  response  categories  being 
constantly  labeled  were  practically  equivalent. 

Order  of  the  Scales 

The  scales  were  ordered  consistently  for  each  factor. 
The  four  evaluation  scales  identified  by  the  selection 
procedure  were  randomly  ordered  in  the  first  four  positions. 
These  were  followed  in  order  by  the  "satisfactory  - 
unsatisfactory"  and  "important  - unimportant"  scales.  There 
was  no  evidence  reported  that  the  order  of  the  scales  influ- 
enced the  ratings.  An  unweighted  factor  score  was  computed 
as  the  average  of  the  four  evaluation  scale  scores . The 
practice  of  following  these  scales  by  the  "satisfactory  - 
unsatisfactory"  scale,  included  as  a measure  of  internal 
consistency,  allowed  a rapid  visual  check  of  the  consistency 
of  a subject's  rating  style.  The  order  also  simplified  the 
data  coding  procedure  for  analysis. 
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The  "important  - unimportant"  scale  was  last  in  order. 
It  was  also  separated  from  the  other  scales  by  the  state- 
ment, "For  me,  this  factor  is."  Since  this  scale  required  a 
different  type  of  judgment  (importance  rather  than  evalu- 
ation), the  statement  set  the  scale  apart  to  provide  a break 
in  the  subject's  evaluative  thought  process.  The  statement 
also  provided  a subtle  reminder  to  the  subject  to  rate  the 
factors  according  to  his  own  feelings  rather  than  an  overall 
judgment  reflecting  the  feelings  of  other  users. 

The  polarity  of  the  scales  was  consistent  with  the 
positive  adjective  located  on  the  left.  Some  investigators 
randomly  alternated  the  positive  term  between  the  left  and 
right  side  (Heise,  1970).  This  practice  was  criticized  by 
Nunnally : 

The  purpose  of  such  reversals  of  polarity  is  to  prevent 
subjects  from  being  influenced  from  scale  to  scale  by 
ratings  made  on  previous  scales.  To  the  extent  that 
this  is  accomplished  by  alternating  directions  of 
scales,  however,  is  not  worth  the  price  in  measurement 
error.  One  frequently  sees  ratings  where  subjects 
apparently  became  confused  by  the  numerous  alternations 
of  scale  directions...  and  rates  a concept  as  both  "very 
good"  and  "very  worthless".  The  weight  of  the  argument 
is  for  keeping  the  scales  pertaining  to  any  factor  all 
pointing  in  the  same  direction  (Nunnally,  1967:5^2). 

Order  of  the  Factors 

The  user  satisfaction  factors  were  randomly  ordered. 
Although  Somer  (1965)  found  no  effect  of  context  or  order  on 
the  subject  ratings,  it  was  felt  that  the  random  order  would 
provide  some  variety  and  a more  interesting  task  to 
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the  respondents. 


Instruction  Set 


Several  sample  instruction  sets  for  semantic  differ- 
ential instruments  were  illustrated  in  Osgood,  Suci,  and 
Tannenbaum  (1957)  and  Nunnally  (1961) . One  criticism  levied 
against  the  sample  instructions  was  that  the  length  and 
elaborateness  were  excessive  for  the  relatively  simple  task 
required  of  the  subject  (Axelrod,  1976).  A simplified 
instruction  set  was  devised  and  provided  to  the  Survey 
Research  Center  of  Arizona  State  University  for  preliminary 
evaluation.  The  final  instruction  set  incorporated  sugges- 
tions made  by  the  Center. 

Assembly 

The  set  of  39  user  satisfaction  factors,  with  the 
respective  scales,  were  assembled  into  a seven  by  eight  and 
a half  inch  booklet  with  the  instructions  on  the  first  page. 
Two  factors  were  placed  on  each  additional  page.  The  factor 
phrases  were  numbered  and  enclosed  in  a box  for  differen- 
tiation from  the  scales.  The  statement  "To  me,  this  factor 
is"  which  preceded  each  "important  - unimportant"  scale  was 
similarly  enclosed  in  a box.  The  use  of  boxes  for  visual 
highlighting  was  recommended  by  Oppenheim  (1966)  and  Babbie 
(1975).  Both  boxes  were  offset  slightly  to  the  left. 

Osipow  and  Grooms  (1962)  found  that  such  offsets  of  the 
response  stimulus  did  not  result  in  a response  bias.  The 
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booklet  was  bound  with  a cover  depicting  a bold  graphic  to 
provide  visual  attractiveness.  A copy  of  the  booklet  pages 
is  included  in  Appendix  4.1. 

PRELIMINARY  EVALUATION 

The  completion  of  the  user  satisfaction  measurement 
instrument  was  followed  by  a preliminary  evaluation.  Four 
user  subjects  were  selected  as  a coarse  representation  of 
the  user  population  of  interest.  These  subjects  completed 
the  questionnaire  and  provided  an  assessment  of  the 
following : 

1.  Clarity  of  the  instructions 

2.  Clarity  of  the  intent  of  the  scales 

3.  Clarity  of  the  factor  phraseology 

4.  Time  to  complete  the  instrument 

The  preliminary  test  determined  that  the  instructions 
were  clear  and  that  there  was  no  confusion  in  interpreting 
the  scales,  with  one  exception.  Two  of  the  test  subjects 
indicated  that  the  scale  "reserved  - unreserved"  for  the 
factor  Confidence  in  systems,  was  ambiguous.  An  alternate 
adjective  pair,  "definite  - uncertain",  identified  from  the 
selection  procedure,  was  acceptable  to  the  subjects  as  a 
replacement.  Several  suggestions  by  the  subjects  on  the 
factor  phraseology  were  incorporated  as  improvements.  The 
time  for  completion  ranged  from  15  minutes  to  25  minutes. 
This  was  within  the  30  minute  upper  bound  that  was  targeted. 
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This  preliminary  evaluation  concluded  the  construction  of 
the  measurement  instrument. 

SUMMARY 

The  semantic  differential  methodology  was  combined  with 
the  user  satisfaction  domain  to  construct  the  measurement 
instrument.  The  specific  adjective  pairs  for  the  factor 
scales  were  selected  in  a two-step,  Delphi  type  procedure. 
Five  university  professors  participated  in  the  selection 
process.  The  adjective  pairs  defined  four  evaluation  scales 
for  each  factor.  The  evaluation  scales  were  augmented  by  an 
internal  consistency  scale  and  an  importance  weighting 
scale . 

The  scales  for  each  factor  were  composed  into  a 
questionnaire  booklet.  The  format  of  the  booklet  was 
designed  to  enhance  the  effectiveness  of  the  measurement 
process.  A preliminary  evaluation  by  four  user  subjects 
confirmed  that  the  instrument  was  clear,  unambiguous,  and 
visually  attractive. 

The  construction  of  the  instrument  established  the 
functional  means  to  measure  user  satisfaction.  The  instru- 
ment was  constructed  under  careful  guidelines  for  the 
content  domain  defined  by  the  methodology  of  Chapter  3.  The 
next  chapter  presents  an  evaluation  of  the  reliability  and 
validity  of  this  instrument. 
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Chapter  5 

EVALUATION  OF  THE  MEASUREMENT  INSTRUMENT 

This  chapter  presents  an  evaluation  of  the  measurement 
instrument.  The  administration  of  the  measurement  instru- 
ment to  a subject  group  and  the  scoring  of  the  results  is 
described.  The  results  are  analyzed  with  the  aid  of  the 
Statistical  Package  for  Social  Scientists  (SPSS)  computer 
programs  for  factor  analysis  and  Pearson  product -moment 
correlation.  These  SPSS  programs  are  augmented  by  data 
reduction  and  analysis  of  variance  programs  developed 
specifically  for  this  research. 

The  evaluation  first  centers  on  the  reliability  prop- 
erty of  the  instrument.  Analysis  of  variance  is  used  to 
show  that  the  estimated  errors  of  measurement  are  insignif- 
icant in  relation  to  the  variance  in  the  scores  attributed 
to  the  measurement  technique.  The  reliabiliy  coefficients 
range  from  .7527  to  .9828. 

The  evaluation  next  focuses  on  the  validity  properties 
of  the  instrument.  Three  categories  of  measurement  validity 
are  identified  as  (1)  content  validity,  (2)  predictive 
validity,  and  (3)  construct  validity.  The  scores  resulting 
from  the  measurement  instrument  are  analyzed  for  charac- 
teristics associated  with  each  of  these  categories  to 
establish  the  overall  validity  properties. 


The  accumulated  results  confirm  that  the  measurement 
instrument  is  both  reliable  and  valid  for  its  intended 
purpose.  Therefore,  the  research  objective  of  user 
satisfaction  measurement  is  effectively  demonstrated. 


ADMINISTRATION  AND  SCORING 
The  32  middle  managers  who  were  interviewed  in  the 
initial  phase  of  the  research  were  selected  as  the  subject 
group  for  evaluation  of  the  measurement  instrument.  The 
selection  of  this  sample  is  described  in  Chapter  3.  The 
representativeness  of  the  subjects  and  their  continued 
commitment  to  participate  in  this  research  were  influencing 
factors  in  this  selection;  however,  the  primary  reason  for 
using  this  particular  group  of  subjects  was  the  existence  of 
self-assessments  of  their  satisfaction  states  at  the  time  of 
the  interviews.  At  the  conclusion  of  each  interview,  each 
subject  was  asked  to  consider  his  total  current  computer 
support  environment  and  to  indicate  his  position  on  the 
seven  interval  satisfaction  scale  shown  in  Figure  5.1. 
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Satisfaction  Scale 
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These  self-assessments  partitioned  the  subjects  into 
groups  of  users  who  considered  themselves  to  be  satisfied  or 
dissatisfied  by  varying  degrees  with  their  current  computer 
support.  This  partitioning  of  users  according  to  their 
satisfaction  states  was  a significant  asset  for  determining 
the  validity  of  the  measurement  instrument. 

Administration  of  the  user  satisfaction  measurement 
instrument  was  initiated  four  to  six  weeks  after  completion 
of  the  interviews.  Twenty-nine  of  the  32  subjects  completed 
and  returned  the  measurement  booklets.  An  informal  follow- 
up to  seven  of  the  subjects  indicated  that  the  time  to 
complete  the  booklet  had  ranged  from  15  to  25  minutes. 

The  measurement  booklets  were  scored  according  to  the 
following  procedures.  Each  of  the  39  factors  of  the  user 
satisfaction  construct  domain  was  rated  on  six  scales.  Four 
of  the  scales  were  for  evaluation  measurement.  The  fifth 
scale,  defined  by  the  adjective  pair  "satisfactory  - 
unsatisfactory",  was  used  for  internal  consistency  purposes 
as  a direct  assessment  of  the  subject's  feeling  about  the 
factor.  The  sixth  scale,  defined  by  "important  - 
unimportant",  was  used  to  weight  the  factor  in  the  overall 
score.  Each  scale  had  seven  interval  response  categories. 
The  integers  +3,  +2,  +1,  0,  -1,  -2,  and  -3  were  assigned  to 
the  intervals  for  the  first  five  scales  with  +3  being 
assigned  to  the  most  favorable  response  interval. 
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The  sixth  scale  was  defined  by  a linear  weighting 
function  to  modify  the  factor  scores  to  reflect  the  saliency 
of  each  factor  in  the  overall  score.  The  weights  assigned 
to  each  interval  were  1.00,  .85,  .70,  .55,  .40,  .25,  and  10. 
The  weight  1.00  was  assigned  to  the  response  "extremely 
important".  The  weights  were  selected  arbitrarily  to 
provide  an  equal  differential  of  .15  (approximately  1/7) 
between  adjacent  intervals. 

The  scale  response  of  subject  k for  adjective  pair  j of 
factor  i is  denoted  by  I^ij . The  raw  score  of  factor  i for 
subject  k was  calculated  as  the  mean  response  for  the  four 
evaluation  scales  and  denoted  FSj^.  Therefore,  the 
potential  range  of  the  factor  scores  was  +3  to  -3  in  .25 
increments.  The  weighted  factor  score  (WFSfci)  was  the 
product  of  the  factor  score  multiplied  by  the  weight 
determined  by  the  sixth  scale  response.  The  factor  satis- 
faction score  (FSSkj.)  was  determined  by  the  fifth  scale 
response.  The  total  satisfaction  score  denoted  TSfc  was  the 
sum  of  all  of  the  unweighted  factor  scores.  The  weighted 
total  satisfaction  score,  denoted  WTSfc,  was  the  sum  of  all 
of  the  weighted  factor  scores.  The  potential  range  of  both 
the  TSk  and  the  WTSfc  was  +117  to  -117  (39  times  +3  and  -3, 
respectively).  Figure  5.2  summarizes  the  scoring  proce- 
dures . 

A computer  program  was  developed  using  the  FORTRAN  IV 
language  to  summarize  the  raw  data  extracted  from  the 


Factor  Score  (FS)*  for  factor  i of  subject  k: 

4 


FSki  = ( l Ikij)/4  i = 1,2,..., 39 

j-1 

Total  Satisfaction  Score  (TS)  for  subject  k: 

39 

TSk  - l FSki 
i=l 

Total  Weighted  Satisfaction  Score  (WTS)  for  subject  k: 

39 

WTSk  - l FSki  Wki 
i=l 

where 

Ikij  = Subject  k's  response  to  scale  j of  factor  i 

Wki  = Importance  weight  for  factor  i as  given  by  subject  k 
Wki  = 1.00  if  Ikif5  = +3 

Wki  = .85  if  Iki,5  - +2 

Wki  = .70  if  Ikif5  = +1 

Wki  = .55  if  Ikif5  = 0 

Wki  = .40  if  Iki,5  = -1 

Wki  = *25  if  Iki,5  = -2 

Wki  = .10  if  Iki,5  = -3 


Figure  5.2 
Scoring  Procedures 
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subject  booklets  and  to  create  input  files  for  subsequent 
analysis.  A copy  of  the  program  is  shown  in  Appendix  5.1. 
The  program  output  included  the  weighted  and  unweighted 
factor  scores  and  total  scores.  The  TSjj  and  WTSk  for  each 
subject  are  shown  in  Table  5.1. 

A comparison  of  the  weighted  and  unweighted  satis- 
faction scores  in  Table  5.1  revealed  a very  close  corres- 
pondence. The  Pearson  product-moment  (PPM)  correlation 
coefficient  for  the  scores  was  .9968.  This  extremely  high 
correlation  coefficient  indicated  that  the  linear 
relationship  between  the  scores  was  almost  perfect. 
Therefore,  the  use  of  a weighting  function  to  alter  the 
factor  scores  did  little  to  increase  the  information 
conveyed  by  the  less  complex  unweighted  satisfaction  score. 

The  individual  booklets  were  examined  to  attempt  to 
explain  this  unexpected  finding.  The  examination  indicated 
that  the  subjects  tended  to  "weight"  the  factors  by  their 
responses  on  the  evaluation  scales.  There  was  a tendency 
for  "important"  factors  to  achieve  more  extreme  evaluation 
ratings  while  the  "unimportant"  factors  were  more  neutrally 
(middle  of  the  scale)  rated.  This  minimized  the  effect  of 
the  weighting  function  of  the  overall  score.  This  propen- 
sity supported  a number  of  researchers  who  contended  that 
differential  weighting  was  unnecessary  (Mobley  and  Locke, 
1970;  Beckwith  and  Lehmann,  1973). 
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Therefore,  the  usefulness  of  the  "important  - 
unimportant"  scale  in  the  measurement  instrument  was 
confined  to  the  identification  of  the  factors  considered 
significant  by  the  individual  subjects.  The  subsequent 
evaluation  was  based  only  on  the  unweighted  factor  and  total 
scores.  The  following  sections  analyze  the  responses  of  the 
29  subjects  to  establish  the  reliability  and  validity 
properties  of  the  measurement  instrument. 

RELIABILITY 

Reliability  is  defined  as  the  relative  absence  of 
measurement  errors.  Thus,  reliability  is  concerned  with  the 
accuracy  of  a measurement  instrument  or,  restated,  the 
extent  to  which  the  obtained  scores  reflect  the  "true" 
scores  of  the  property  being  measured.  Since  the  "true" 
scores  are  unknown  in  any  set  of  obtained  scores,  the  error 
scores  are  also  unknown  and  must  be  estimated.  This 
estimate  is  based  upon  the  assumption  that  the  errors  are 
independent  and  random.  They  are  assumed  to  be  uncorrelated 
to  the  true  scores.  Therefore,  the  total  variance  in  any 
set  of  measurement  scores  is  composed  of  "true"  variance  and 
error  variance. 

The  reliabiliiy  of  the  user  satisfaction  measurement 
instrument  was  determined  by  the  proportion  of  variance  in 
the  factor  scores  that  was  attributable  to  measurement 
error.  For  any  set  of  scores  obtained  by  the  instrument, 
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the  total  variance  was  composed  of  the  variance  from 

individual  subject  differences  and  the  error  variance.  An 

f analysis  of  variance  (ANOVA)  approach  was  used  to  calculate 

I $ 

the  total  variance  and  the  different  variance  components  for 
each  factor  score.  A sample  ANOVA  table  is  illustrated  in 
Figure  5.3.  Readers  unfamiliar  with  analysis  of  variance 
are  referred  to  Walpole  and  Myers  (1972)  or  Hicks  (1973), 
among  others.  The  reliability  coefficient  for  each  factor 
was  computed  by  the  equation: 

rtt  = 1 - (Ve  / Vsub)  (5.1) 

where 

rtt  = the  reliability  coefficient 
Vsub  = variance  due  to  subject  difference 

IVe  = error  or  residual  variance 

This  method  of  estimating  reliability  has  been  widely 

I accepted  (Thorndike,  1967;  Nunnally,  1967;  Kerlinger,  1973; 

Summey,  1974).  Kerlinger  explained  the  interpretation  of 
the  coefficient: 

The  reliability  coefficient  is  a coefficient  of 
determination.  Theoretically,  it  tells  how  much 
variance  of  the  total  variance  of  a measured  variable  is 
"true"  variance.  If  we  had  the  "true"  scores  and  could 
correlate  them  with  the  scores  of  the  measured  variable 
and  square  the  resulting  coefficient  of  correlation,  we 
would  obtain  the  reliability  coefficient  (Kerlinger, 
1973:451) . 

A separate  reliability  coefficient  was  calculated  for 
each  of  the  39  factors  which  comprise  the  user  satisfaction 
domain.  This  was  required  because  each  factor  represented  a 
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Factor  39 


SUBJECTS 

SCALE 

TOTALS 

1 

2 

3 

4 

1 

7.00 

7.00 

7.00 

7.00 

28.00 

2 

7.00 

7.00 

7.00 

7.00 

28.00 

3 

4.00 

4.00 

4.00 

4.00 

16.00 

4 

5.00 

6.00 

5.00 

5.00 

21.00 

5 

1.00 

1.00 

2.00 

2.00 

6.00 

6 

4.00 

4.00 

4.00 

4.00 

16.00 

7 

2.00 

2.00 

3.00 

3.00 

10.00 

8 

2.00 

4.00 

4.00 

4.00 

14.00 

9 

2.00 

2.00 

2.00 

2.00 

8.00 

10 

5.00 

5.00 

5.00 

4.00 

19.00 

11 

3.00 

3.00 

2.00 

3.00 

11.00 

12 

6.00 

6.00 

5.00 

6.00 

23.00 

13 

5.00 

3.00 

5.00 

4.00 

17.00 

14 

2.00 

2.00 

3.00 

3.00 

10.00 

15 

3.00 

5.00 

5.00 

5.00 

18.00 

16 

6.00 

6.00 

7.00 

6.00 

25.00 

17 

2.00 

2.00 

2.00 

2.00 

8.00 

18 

1.00 

1.00 

1.00 

1.00 

4.00 

19 

1.00 

1.00 

1.00 

1.00 

4.00 

20 

1.00 

1.00 

1.00 

1.00 

4.00 

21 

1.00 

1.0J 

1.00 

1.00 

4.00 

22 

5.00 

5.00 

5.00 

5.00 

20.00 

23 

3.00 

3.00 

3.00 

3.00 

12.00 

24 

2.00 

2.00 

3.00 

3.00 

10.00 

25 

2.00 

2.00 

2.00 

2.00 

8.00 

26 

6.00 

5.00 

5.00 

5.00 

21.00 

27 

2.00 

2.00 

3.00 

4.00 

11.00 

28 

4.00 

5.00 

4.00 

4.00 

17.00 

29 

6.00 

6.00 

7.00 

6.00 

25.00 

TOTAL 

100.00 

103.00 

108.00 

107.00 

Source 

SS 

DF 

MS 

TOTAL 

409.76 

115.00 

SCALES 

1.41 

3.00 

.47 

SUBJECTS 

388.26 

28.00 

13.87 

ERROR 

20.09 

84.00 

.24 

Figure  5.3 
Sample  ANOVA  Table* 

*The  scale  values  include  a constant  of  +4.0  which  was  added 
to  eliminate  negative  numbers  during  the  input  process. 
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unique,  independent  measurement  problem.  A computer  program 
was  developed  using  FORTRAN  IV  to  perform  the  required 
analysis  of  variance  and  computation  of  the  reliability 
coefficients  for  each  factor.  Additional  factor  statistics 
were  also  computed.  The  program  listing  and  a sample  output 
is  shown  in  Appendix  5.2.  The  reliability  coefficients  are 
shown  in  Table  5.2. 

The  reliabilty  coefficients  obtained  were  exceptionally 
high.  Thirty-two  of  the  39  factors  had  a coefficient  that 
exceeded  .90.  The  minimum  coefficient  was  .7527.  These 
results  indicated  that  very  little  of  the  variation  in  the 
factor  scores  was  attributable  to  measurement  error. 

Rather,  the  differences  were  a result  of  measured  differ- 
ences between  the  subjects.  This  level  of  reliability  was 
very  favorably  comparable  to  reliability  coefficients 
developed  in  similar  research  (Heise,  1969;  Summey,  1974). 
Reliability  was  a necessary  but  not  sufficient  condition  for 
validity  (Nunnally,  1967) . The  next  section  discusses  the 
evaluation  of  the  measurement  instrument  for  validity. 

VALIDITY 

The  validity  of  a measurement  instrument  is  determined 
by  the  extent  to  which  it  measures  what  it  is  supposed  to 
measure.  While  the  theory  of  measurement  error  is  well 
developed  and  straightforward  for  establishing  reliability, 
there  is  no  well  defined  process  for  documenting  the 
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Table  5.2 

Reliability  Coefficients  by  Factor  (Fj.) 


t 

Factor 

i 

Reliability 

Coefficient 

Factor 

i 

Reliability 

Coefficient 

r 

1 

.9295 

21 

.9188 

2 

.8469 

22 

.9446 

3 

.9347 

23 

.9431 

\ * 

4 

.7527 

24 

.9575 

5 

.9826 

25 

.8614 

6 

.9237 

26 

.8327 

7 

.9365 

27 

.9133 

8 

.9662 

28 

.9630 

y 

9 

.9008 

29 

.9624 

10 

.8902 

30 

.9651 

11 

.9143 

31 

.8125 

12 

.8698 

32 

.9297 

13 

.9324 

33 

.9542 

14 

.9787 

34 

.9495 

15 

.9256 

35 

.9688 

16 

.9286 

36 

.9683 

17 

.9500 

37 

.9602 

18 

.9417 

38 

.9724 

19 

.9733 

39 

.9828 

I 

20 

.9060 
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validity  of  a measurement  instrument  designed  for  psycho- 
metric purposes.  Rather,  validity  is  viewed  as  a property 
determined  by  logical  as  well  as  statistical  arguments. 

Three  traditional  categories  of  measurement  validity 
have  evolved,  however,  that  characterize  the  different 
aspects  of  ■»  alidity  as  well  as  the  approaches  that  have  been 
pursued  to  demonstrate  this  property.  These  are  (1)  content 
validity,  (2)  predictive  validity,  and  (3)  construct 
validity  (Ghiselli,  1964;  Nunnally,  1967).  The  reader  is 
referred  to  Chapter  2 for  a fuller  discussion  of  these 
categories.  The  user  satisfaction  measurement  instrument 
was  evaluated  for  each  of  the  three  types  of  validity. 

Content  Validity 

Content  validity  is  determined  from  the  content  and 
operations  of  the  measurement  instrument.  It  is  sometimes 
referred  to  as  "face  validity"  (Babbie,  1975).  Content 
validity  implies  that  the  content  of  the  construct  or 
attribute  being  measured  has  been  adequately  sampled  from 
the  total  construct  domain  and  that  the  operations  to 
measure  this  content  are  logically  sound  (Ghiselli,  1964). 

The  content  of  the  user  satisfaction  measurement 
instrument  was  designed  to  include  all  important  factors 
affecting  a user's  satisfaction  with  the  computer  support 
provided  to  him  within  the  organization.  The  methodology  to 
identify  those  factors  and  to  test  for  the  completeness  of 
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the  factor  list  is  presented  in  Chapter  3.  The  probability 
of  any  factor  considered  important  by  a user  being  included 
in  the  measurement  instrument  exceeded  .90  in  a test  of 
hypotheses  at  the  .01  level  of  significance.  This  was 
compelling  evidence  that  the  domain  of  the  user  satisfaction 
construct  had  been  more  than  "adequately"  sampled  in  the 
construction  of  the  measurement  instrument. 

The  semantic  differential  methodology  was  used  to 
establish  the  operations  for  measuring  this  content.  An 
extensive  evaluation  of  the  appropriateness  of  this 
methodology  selection  is  presented  in  Chapter  2.  The 
careful  development  of  the  measurement  instrument  itself  is 
described  in  Chapter  4.  It  was  felt  that  this  deliberate 
and  prudent  approach  to  the  functionalization  of  the 
measurement  process  justified  a validity  claim;  however, 
this  was  ultimately  a subjective  judgment.  Therefore,  the 
response  structures  of  the  29  subjects  who  were  administered 
the  instrument  were  examined  for  further  evidence  of  content 
validity  that  was  not  based  upon  subjective  judgments. 

Three  characteristics  of  the  response  structures  were 
considered  indicative  of  content  validity.  The  first  was 
homogeneity  or  internal  consistency  of  the  individual  factor 
scales.  Scales  which  purport  to  measure  the  same  thing 
should  be  positively  correlated  (or  homogeneous) . The 
second  was  unidimensionality  of  the  scales  within  each 
factor.  Since  the  satisfaction  construct  was  considered  to 
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be  composed  of  only  the  Evaluation  dimension  of  connotative 
feeling,  the  factor  scales  should  reflect  only  one  dimen- 
sion. Finally,  the  third  characteristic  was  scale  discrim- 
ination between  "satisfied"  and  "dissatisfied"  subjects. 

The  Pearson  product-moment  (PPM)  correlation  coef- 
ficients were  calculated  for  each  pair  of  scales  for  each 
factor  to  determine  the  homogeneity  of  the  measurement 
instrument.  The  correlation  coefficients  are  shown  in  Table 
5.3.  A significance  test  for  the  correlation  coefficient  r 
was  derived  from  the  use  of  the  Student's  t distribution 
with  N-2  degrees  of  freedom  for  the  quantity 

r [ (N-2)  / (1-r 2)  ]* 

(Nie,  Hull,  Jenkins,  Steinbrenner , and  Bent,  1970).  All  but 
8 of  the  156  coefficients  were  significant  at  the  .01  level. 
All  but  one  of  the  coefficients  were  significant  at  the  .05 
level.  It  was  concluded  that  the  measurement  instrument  was 
internally  consistent. 

Factor  analysis  was  used  to  examine  the  scales  for 
unidimensionality.  Factor  analysis  is  a mathematical  method 
of  discovering  patterns  in  the  variations  of  the  values  of 
several  variables.  This  is  accomplished  by  the  generation 
of  artificial  dimensions,  termed  "factors"  (these  should  not 
be  confused  with  the  factors  in  the  measurement  instrument), 
which  correlate  highly  with  several  of  the  variables  and 
that  are  independent  of  one  another  (Babbie,  1975).  The 
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Table  5.3 


Correlation  Coefficients  Betv/een  Factor  Scales 


Factor 

i 

1,2 

1,3 

Scale 

1,4 

Pairs  i, 
2,3 

j 

2,4 

3,4 

1 

.9279 

.6746 

.7761 

.7466 

.7487 

.7646 

2 

.8952 

.3489 

.5149 

.4537 

.6744 

.5678 

3 

.7349 

.8404 

.7254 

.8468 

.7406 

.8031 

4 

.6750 

.3856 

.6733 

.0156 

.3821 

.6079 

5 

.9555 

.9486 

.9264 

.9355 

.9486 

.8985 

6 

.7783 

.7098 

.8521 

.6456 

.6769 

.8554 

7 

.9332 

.9322 

.6680 

.9173 

.7126 

.7721 

8 

.9624 

.8967 

.8287 

.8934 

.8227 

.8797 

9 

.8705 

.4842 

.6577 

.6271 

.7734 

.7886 

10 

.4179 

.9222 

.8136 

.4993 

.4802 

.8951 

11 

.7894 

.6652 

.7598 

.9061 

.6197 

.6237 

12 

.8963 

.7263 

.4492 

.7839 

.4257 

.4773 

13 

.8600 

.7619 

.6914 

.7634 

.6849 

.8994 

14 

.9558 

.9449 

.8683 

.9616 

.8812 

.9200 

15 

.8645 

.8262 

.6652 

.8539 

.6147 

.7059 

16 

.6421 

.8500 

.8320 

.6796 

.7701 

.8140 

17 

.7965 

.8088 

.8635 

.8241 

.8217 

.8577 

18 

.9009 

.7234 

.8597 

.8353 

.8114 

.8295 

19 

.9135 

.9135 

.8655 

.9235 

.8765 

.9304 

20 

.6823 

.9090 

.7746 

.6959 

.4992 

.6796 

21 

.7099 

.6979 

.6906 

.8444 

.7639 

.7802 

22 

.9527 

.6782 

.9305 

.7002 

.9052 

.7193 

23 

.8898 

.8994 

.7521 

.8537 

.7345 

.7698 

24 

.9541 

.8576 

.8107 

.8435 

.8449 

.7965 

25 

.8451 

.8734 

.4778 

.7432 

.4517 

.4543 

26 

.5979 

.5286 

.5727 

.4214 

.4764 

.8175 

27 

.5964 

.9083 

.8551 

.5392 

.6143 

.8649 

28 

.9560 

.8594 

.8062 

.9142 

.8554 

.8436 

29 

.8827 

.8950 

.9101 

.8309 

.8262 

.8856 

30 

.8564 

.8276 

.8180 

.9040 

.9151 

.9441 

31 

.4281 

.3481 

.6408 

.6854 

.5467 

.5375 

32 

.7332 

.7173 

.6630 

.8269 

.7593 

.9032 

33 

.8599 

.7780 

.9233 

.8391 

.8650 

.7806 

34 

.8061 

.8628 

.7516 

.8958 

.7837 

.8849 

35 

.8902 

.8964 

.9592 

.8692 

.8616 

.8810 

36 

.7982 

.8859 

.8809 

.8723 

.8792 

.9839 

37 

.8632 

.8632 

.9121 

.8034 

.8175 

.8863 

38 

.9374 

. 8406 

.9355 

.8740 

.9142 

.8952 

39 

.9327 

.9257 

.9175 

.9336 

.9474 

.9658 

V 
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number  of  factors  that  are  generated  is  a function  of  the 
number  of  "clusters"  that  exist  in  the  variables.  One 
output  of  factor  analysis  is  the  "factor  loadings"  which  are 
the  correlations  between  each  factor  and  each  variable. 
Factor  analysis  is  actually  a generic  term  that  subsumes  a 
variety  of  techniques.  The  statistical  basis  is  suffi- 
ciently complex  that  the  interested  reader  is  referred  to 
Harmon  (1967)  or  Nie,  Hull,  Jenkins,  Steinbrenner , and  Bent 
(1970)  for  a more  complete  discussion. 

The  scale  values  for  each  user  satisfaction  factor  were 
input  as  variables  to  the  SPSS  factor  analysis  program.  A 
sample  output  is  shown  in  Appendix  5.3.  With  one  exception, 
the  scale  values  were  defined  by  only  one  dimension.  The 
exception  was  the  factor  Organizational  Competition  with  the 
EDP  Unit  (F4)  which  had  two  dimensions.  The  second  dimen- 
sion for  this  factor  was  due  to  the  third  scale  which  did 
not  correlate  strongly  with  the  other  evaluative  scales. 

This  exception  was  not  considered  to  be  significant  because 
the  factor  was  ranked  next  to  last  in  importance  by  the 
subjects.  The  single  factor  loadings  are  shown  in  Table 
5.4.  These  results  confirmed  the  unidimensionality  of  the 
scales  in  the  measurement  instrument. 

The  scales  discrimination  between  "satisfied"  and 
"dissatisfied"  users  was  next  examined  on  a factor  by  factor 
basis.  The  29  subjects  were  dichotomized  into  a satisfied 
or  dissatisfied  group  based  upon  the  sign  (plus  or  minus, 


Factor 

i 

1 

2 

Scale 

3 

4 

1 

.9190 

.9424 

.8026 

.8550 

2 

.8076 

.9983 

.5276 

.7207 

3 

.8590 

.8705 

.9687 

.8405 

4 

.8827 

.6402 

.5637 

.8173 

4* 

.1897 

.5775 

.5960 

.2463 

5 

.9791 

.9838 

.9552 

.9507 

6 

.9104 

.7791 

.8413 

.9433 

7 

.9452 

.9568 

.9868 

.7458 

8 

.9639 

.9591 

.9509 

.8806 

9 

.7967 

.9391 

.7258 

.8918 

10 

.9055 

.4974 

.9993 

.9054 

11 

.8600 

.9398 

.8664 

.7484 

12 

.9127 

.9489 

.8297 

.4996 

13 

.8694 

.8670 

.9347 

.8551 

14 

.9618 

.9772 

.9910 

.9114 

15 

.9183 

.9114 

.9327 

.7192 

16 

.8938 

.7657 

.9014 

.9403 

17 

.9012 

.8863 

.9125 

.9417 

18 

.9140 

.9456 

.8604 

.9177 

19 

.9407 

.9514 

.9799 

.9309 

20 

.9908 

.7048 

.9326 

.7466 

21 

.7835 

.9084 

.9118 

.8573 

22 

.9700 

.9641 

.7253 

.9581 

23 

.9551 

.9184 

.9423 

.8014 

24 

.9617 

.9719 

.8904 

.8677 

25 

.9803 

.8581 

.8823 

.5077 

26 

.7169 

.6094 

.8161 

.8815 

27 

.9506 

.6232 

.9307 

.9267 

28 

.9374 

.9933 

.9319 

.8748 

29 

.9723 

.8944 

.9312 

.9373 

30 

.8691 

.9577 

.9599 

.9614 

31 

.6217 

.7728 

.7236 

.8022 

32 

.7717 

.8792 

.9636 

.8925 

33 

.9389 

.9366 

.8606 

.9431 

34 

.8720 

.9063 

.9966 

.8730 

35 

.9810 

.9142 

.9270 

.9580 

36 

.8945 

.8868 

.9894 

.9903 

37 

.9610 

.8794 

.9150 

.9495 

38 

.9571 

.9620 

.9030 

.9720 

39 

.9499 

.9688 

.9752 

.9783 

* Loading  on  second  dimension 
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respectively)  of  their  total  factor  score  (average  of  the 
four  scales) . The  scale  means  for  each  group  were  then 
calculated.  The  range  for  each  scale  was  the  difference 
between  the  two  group  means.  The  range  was  indicative  of 
the  discriminatory  power  of  the  scale.  The  scale  ranges  are 
shown  in  Table  5.5.  Ninety-seven  of  the  156  scales  had  a 
range  of  more  than  3 intervals.  The  minimum  range  for  any 
scale  was  1.67  intervals.  Therefore,  each  of  the  scales  was 
considered  to  be  sufficiently  discriminating. 

Therefore,  the  response  structures  of  the  subjects 
exhibited  the  desired  characteristics  of  homogeneity,  uni- 
dimensionality, and  discrimination.  When  considered  with 
the  methodologies  used  to  define  the  content  of  the  measure- 
ment instrument  and  to  measure  that  content,  the  claim  of 
content  validity  was  deemed  to  be  justified. 


♦ 


Predictive  Validity 


Predictive  validity  is  traditionally  signified  by  the 
correlation  between  the  measurement  scores  and  some 


independent  criterion  that  is  assumed  to  represent  the 
"true"  state  of  the  attribute  being  measured.  The  useful- 
ness of  predictive  validity  measures  has  often  been  limited 
by  the  absence  of  an  appropriate  independent  criterion. 

This  limitation  has  been  especially  acute  for  psychometric 
measurement  instruments  designed  to  measure  "how  much"  of  an 
attribute  exists  in  a subject  (Nunnally,  1967). 


■ 

[ | 

1 

[ A 

• 

. i 

16 

3 

♦ 

Table 

5.5 

Scale  Range  Between  Means 

Factor 

Scale 

i 

1 

2 

3 

4 

i 

1 3.46  3.33  2.67  2.2 

9 

2 2.61  2.96  2.08  2.6 

5 

3 2.95  3.43  3.24  3.3 

8 

1 

4 1.9 

8 1.6 

7 2.0 

9 2.0 

0 

5 4.0 

3 3.5 

3 

i 

6 4.5 

0 4.5 

4 3.8 

7 3.8 

7 

7 2.1 

2 2.5 

9 

8 4.3 

8 3.9 

9 4.0 

4 3.3 

8 

■chi 

1 2.5 

0 3.2 

2 

1 

0 2.8 

8 2.0 

5 2.8 

3 2.4 

8 

1 

1 3.7 

5 4.1 

7 3.3 

7 2.4 

5 

■ 

1 

2 3.2 

4 2.7 

5 2.2 

6 1.8 

1 

1 

3 3.5 

9 3.3 

0 3.6 

6 2.9 

0 

i 

2 3.2 

3 2.8 

0 

1 

5 2.5 

0 3.2 

5 2.7 

3 1.8 

7 

1 

6 3.3 

9 2.2 

8 3.2 

3 3.4 

8 

1 

1 

8 3.7 

Q 1ft 

0 3.0 

n o l 

8 

Q 

1 

1 

U J • 7 

9 3.7 

/ ^ . O 

5 3.9 

7 1.0 

6 3.4 

O 

7 

2 

0 2.9 

3 2.1 

2 2.3 

1 1.8 

1 

2 

1 2.2 

9 3.8 

7 

2 

2 3.3 

4 3.2 

9 2.7 

0 3.2 

4 

2 

3 3.1 

3 3.1 

3 2.5 

0 3.2 

5 

2 

4 3.7 

6 3.5 

5 3.3 

3 3.0 

5 

2 

5 3.8 

9 4.1 

1 3.8 

6 3.5 

7 

2 

6 2.3 

7 1.9 

7 2.5 

1 2.3 

1 

2 

7 4.2 

7 2.5 

5 3.8 

6 3.1 

9 

r 

2 

1 3.3 

0 3.1 

2 2.7 

4 

2 

9 3.4 

6 3.6 

7 2.4 

6 2.6 

7 

3 

0 2.8 

2 3.0 

5 3.4 

5 3.1 

2 

3 

1 2.7 

0 2.0 

8 2.1 

5 2.1 

0 

3 

2 2.5 

5 2.7 

8 3.1 

0 3.4 

5 

3 

3 3.3 

9 2.7 

2 3.0 

6 

3 

1 

4 2.8 

5 

3 

5 3.2 

3 4.1 

5 3.1 

5 3.0 

8 

3 

6 3.2 

4 3.2 

4 3.5 

8 3.6 

2 

3 

7 3.2 

4 2.7 

0 2.9 

7 2.9 

4 

3 

8 3.7 

0 3.7 

5 3.3 

5 3.3 

8 

3 

9 3.6 

2 3.5 

7 3.3 

9 
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There  was  no  known  independent  criterion  external  to 
this  research  that  specifically  addressed  the  satisfaction 
of  a user,  either  by  factor  or  total  score,  that  could  be 
used  to  evaluate  the  user  satisfaction  measurement  instru- 
ment. Two  data  elements,  however,  were  collected  during 
this  research  that  were  conducive  to  the  predictive  validity 
evaluation.  The  first  was  the  self-assessment  by  each 
subject  of  his  total  satisfaction  state  at  the  time  of  the 
initial  interview.  These  assessments  were  made  four  to  six 
weeks  prior  to  the  administration  of  the  measurement 
instruments.  Therefore,  the  subject  group  was  partitioned 
into  "known"  groups  along  a total  satisfaction  scale. 

The  self-assessments  and  the  measured  satisfaction 
scores  of  the  subjects  are  shown  in  Table  5.6.  The  subjects 
are  grouped  by  organization.  A graph  of  the  scores  is 
illustrated  in  Figure  5.4  with  the  horizontal  axis  repre- 
senting the  self-assessment  and  the  vertical  axis  repre- 
senting the  measured  score.  The  data  points  are  identified 
by  subject.  The  PPM  correlation  coefficient  for  the  two 
scores  was  .7899.  This  was  significant  at  the  .001  level. 

It  was  felt  that  this  correlation  was  quite  high  considering 
that  the  self-assessment  scores  could  assume  only  one  of 
seven  values.  This  correlation  between  the  measured  scores 
and  an  independent  criterion  of  user  satisfaction  was 
indicative  of  predictive  validity. 
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Table  5.6 

Subject  Satisfaction  Scores 


Subject 

Self-Assessed 

Score 

Measured 

Score 

Organizational 

Mean 

A1 

+3 

111.75 

68.06 

A2 

+2 

49.75 

A3 

+3 

69.25 

A4 

+2 

41.50 

61 

+2 

15.25 

15.25 

Cl 

+2 

48.50 

14.19 

C3 

-1 

20.00 

C4 

-2 

-4.75 

C5 

-1 

-7.00 

D1 

+1 

54.25 

48.58 

D2 

+1 

31.25 

D3 

+2 

60.25 

El 

+2 

63.75 

59.06 

E2 

+2 

57.75 

E3 

+2 

31.75 

E4 

+3 

83.00 

FI 

-2 

-38.00 

-38.45 

F2 

-2 

-46.75 

F3 

-2 

-27.75 

F4 

+1 

-15.50 

F5 

-3 

-64.25 

G1 

+2 

21.50 

10.50 

G2 

+1 

18.50 

G3 

+1 

17.75 

G4 

+2 

-15.75 

HI 

+2 

60.75 

51.50 

H2 

-2 

33.75 

H3 

+2 

55.25 

H4 

+2 

56.25 

This  evidence  was  further  supported  by  the  linearity 
exhibited  by  the  subject  scores.  After  partitioning  the 
groups  according  to  their  self-assessments,  the  mean 
measured  scores  were  calculated  for  each  group.  The 
response  group  means  are  shown  in  Table  5.7. 

The  second  data  element  for  evaluating  predictive 
validity  was  provided  in  the  measurement  instrument  itself. 

A separate  scale  defined  by  the  adjective  pair  "satisfactory 
- unsatisfactory"  was  included  for  each  factor.  This  scale 
was  considered  a "direct"  assessment  of  the  subject's 
satisfaction  with  the  individual  factors.  This  direct 
assessment  should  correlate  highly  with  the  factor  score 
determined  by  the  four  evaluation  scales.  The  independence 
of  this  criterion  was  subject  to  question  because  it  was 
collected  at  the  same  time  and  in  the  same  manner  as  the 
measured  scores;  therefore,  the  absence  of  a high  corre- 
lation would  have  been  more  significiant  as  a negative 
indication  of  predictive  validity  than  the  converse 
condition.  However,  as  the  results  shown  in  Table  5.8 
indicate,  the  correlations  were  extremely  high  for  the  two 
variables.  The  PPM  correlation  coefficients  ranged  from 
.7454  to  .9719  and  were  significant  at  the  .001  level. 

The  results  accumulated  in  this  evaluation  of  the 
subject  self-assessments  and  the  direct  factor  scores 
sustained  the  predictive  validity  property  of  the  user 
satisfaction  measurement  instrument.  Since  the  subject 
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Table  5.7 

Response  Group  Mean  Scores 


Self-Assessment 
Response  Group 


Measured  Mean 
Score  (subjects) 


Extremely  Satisfied 

88, 

.00 

(3) 

Quite  Satisfied 

42, 

.04 

(13) 

Slightly  Satisfied 

21. 

25 

(5) 

Slightly  Dissatisfied 

6. 

.50 

(2) 

Quite  Dissatisfied 

-16, 

.70 

(5) 

Extremely  Dissatisfied 

-64, 

.25 

(1) 
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Table  5.8 

Satisfaction  Scale  and  Factor  Score  Correlation 


Factor 

i 

Correlation 

Coefficient 

Factor 

i 

Correlation 

Coefficient 

1 

.9273 

21 

.8044 

2 

.8769 

22 

.9283 

.9468 

23 

.9700 

4 

.7454 

24 

.9443 

5 

.9699 

25 

.9099 

6 

.9207 

26 

.8880 

7 

.8237 

27 

.8832 

8 

.9719 

28 

.9003 

9 

.9659 

29 

.8434 

10 

.8666 

30 

.9267 

11 

.8967 

31 

.8860 

12 

. 9169 

32 

.9279 

13 

.9130 

33 

.8752 

14 

.9656 

34 

.8052 

15 

.9469 

35 

.9585 

16 

.9252 

36 

.9673 

17 

.9490 

37 

.9530 

18 

.9368 

38 

.9359 

19 

.9158 

39 

.9218 

20 

.8451 
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group  had  participated  throughout  the  research,  an  attempt 
was  made  to  develop  yet  another  independent  criterion  for 
comparing  the  measured  scores.  The  vice  president  of  a 
local  computer  company,  which  sells  and  services  hardware 
and  software  systems  for  small  businesses,  was  contacted  and 
asked  to  identify  six  customers  in  each  of  three  categories 
according  to  the  instructions  shown  in  Appendix  5.4.  The 
categories  were  customers  who  were  (1)  extremely  satisfied, 
(2)  extremely  dissatisfied,  and  (3)  slightly  satisfied  or 
dissatisfied.  He  was  asked  to  identify  only  those  customers 
whom  he  felt  confident  of  placing  in  the  correct  category. 
This  request  proved  to  be  much  more  difficult  than  presumed. 
Although  six  customers  were  named  for  each  group,  some  were 
named  by  the  vice  president  with  trepidation  that  they  were 
mislabeled. 

Each  of  the  users  identified  in  this  manner  was  asked 
to  complete  the  measurement  booklet.  Eleven  of  the  users 
completed  and  returned  the  booklet.  The  subject  scores  by 
response  category,  shown  in  Table  5.9,  were  less  than 
conclusive.  Although  the  two  highest  and  the  two  lowest 
user  scores  were  associated  with  the  expected  response 
categories,  the  other  scores  did  not  fall  neatly  within 
predicted  clusters.  The  mean  scores  of  the  response 
categories  did,  however,  rank  in  the  expected  order.  Only 
two  of  the  subjects,  S3  and  D4,  achieved  scores  signif- 
icantly different  than  their  category  would  indicate.  This 
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Table  5.9 
Measured  Scores 


Response  Category 

Subj  ect 

Score 

Category  Mean 

Extremely  Satisfied 

S2 

82.00 

S3 

37.50 

S4 

62.75 

72.75 

S5 

66.00 

S6 

115.50 

Slightly  Satisfied 

Ml 

65.50 

67.67 

or  Dissatisfied 

M5 

74.25 

M6 

63.25 

Extremely  Dissatisfied 

Dl 

-4.25 

-9.50 

D4 

59.00 

D6 

-64.25 

t 
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anomaly  was  probably  the  result  of  misidentification  during 
the  selection  process. 

The  user  satisfaction  measurement  instrument  was 
subjected  to  three  approaches  to  determine  predictive 
validity.  It  was  felt  that  this  property  of  the  measurement 
instrument  was  confirmed  by  the  combined  results. 

Construct  Validity 

Construct  validity  is  based  upon  the  behavior  of  the 
measurement  scores  conforming  to  expected  relationships  with 
the  construct  network  of  variables.  Cronbach  and  Meehl 
(1955:300)  state  that  "construct  validity  is  possible  only 
when  some  of  the  statements  in  the  network  lead  to  predicted 
relations  among  observables."  This  type  of  restriction 
is  complicated  by  the  fact  that  a "satisfaction"  is  not  ever 
seen  and  is  therefore  not  an  observable.  Surrogate 
criteria,  such  as  the  number  of  complaints  or  late  bill 
payments,  are  themselves  subject  to  questions  of  validity. 
Consequently,  it  is  ordinarily  necessary  to  evaluate 
construct  validity  by  integrating  evidence  from  the  many 
different  sources. 

The  only  observables  that  were  available  to  evaluate 
the  user  satisfaction  measurement  instrument  for  construct 
validity  were  the  subject  scores.  The  foundation  for 
construct  validity  was  embodied  in  the  evaluation  findings 
for  content  and  predictive  validity.  The  definition  of  the 


user  satisfaction  construct  and  the  subsequent  definition  of 
the  content  and  the  measurement  process  (content  validity) 
established  certain  expectations  regarding  the  response 
structures.  The  subject  responses  did  exhibit  the  expected 
properties  of  internal  consistency,  unidimensionality,  and 
scale  discrimination.  Furthermore,  the  subject  responses 
correlated  highly  with  the  responses  anticipated  on  the 
basis  of  external  criteria  (predictive  validity) . These 
relationships  followed  a pattern  that  implied  construct 
validity. 

Additional  indications  of  construct  validity  were 
contained  in  the  subject  responses.  In  developing  the 
content  of  the  user  satisfaction  measurement  instrument, 
importance  had  been  attributed  to  the  factors  on  the  basis 
of  literature  references  and  subject  mentions  during 
interviews.  The  subject  responses  to  the  measurement 
instrument  confirmed  this  implied  importance.  The  mean 
response  for  each  factor's  "important  - unimportant"  scale 
is  shown  in  Table  5.10  in  descending  order  within  the  scale 
response  boundaries.  Each  of  the  factors  was  accorded  a 
positive  importance  mean.  The  relatively  low  means  of  the 
Language  (F^0) » Organizational  competition  (F4) , and  Charge- 
back  (F -7)  factors  were  presumed  to  be  a result  of  the  lack 
of  generality  of  the  factors.  For  example,  only  one  of  the 
subject  organizations  utilized  a charge-back  method  of 
payment  for  services.  Nevertheless,  every  factor  was  ranked 
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Table  5.10 

Importance  Scale  Means  by  Factor 


Scale  Mean 


+3.00  EXTREMELY  IMPORTANT* 


Factor  (F^) 


2.59 

2.48 

2.38 

2.38 

2.28 

2.17 

2.14 

2.14 

2.07 

2.07 

2.07 

2.03 

2.03 

2.03 


Accuracy 

Reliability 

Timeliness 

Relevancy 

Confidence  in  systems 

Communication  with  EDP  staff 

Error  recovery 

Attitude  of  the  EDP  staff 

Relationship  with  the  EDP  staff 

Schedule  of  products  and  services 

Technical  competence  of  the  EDP  staff 

Response/ turnaround  time 

Job  effects 

Completeness 


+2.00  QUITE  IMPORTANT* 


2.00 

1.97 

1.97 

1.97 

1.93 

1.90 

1.86 

1.79 

1.79 

1.72 

1.69 

1.66 

1.62 

1.59 

1.55 

1.38 

1.38 

1.38 

1.34 

1.24 

1.21 

1.03 


Time  required  for  new  development 

Processing  of  change  requests 

Currency 

Flexibility 

Perceived  utility 

Expectations 

Format  of  output 

Precision 

Do cumentation 

Mode  of  interface 

Feeling  of  participation 

Convenience  of  access 

Security  of  data 

Priorities  determination 

Integration  of  systems 

Understanding  of  systems 

Top  management  involvement 

Feeling  of  control 

Volume  of  output 

Vendor  suppport 

Degree  of  training 

Organizational  position  of  EDP  function 


Scale  i Factor  (F^) 

> 

+1.00  SLIGHTLY  IMPORTANT* 

.93  10  Language 

.31  4 Organizational  competition  with  EDP  unit 

. 14  7 Charge-back  method  of  payment  of  services 

I ============================================================== 

* Response  boundary 
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"extremely  important"  by  at  least  one  subject. 

The  order  of  the  factors  rated  most  important  overall 
also  had  logical  appeal.  The  top  rated  four  factors  seemed 
to  indicate  the  importance  of  computer  information  products 
and  services  that  were  accurate  (F2g)  . reliable  (F^) , 
timely  (Fg),  and  relevant  (F25)  to  the  user’s  needs.  It 
appeared  intuitive  that  these  four  factors  formed  the  crux 
of  the  satisfaction  construct. 

The  factors  causing  dissatisfaction  were  also 
revealing.  The  number  of  subjects  expressing  a negative, 
hence  dissatisfied,  score  for  each  factor  is  shown  in  Table 
5.11.  The  factors  identified  by  an  asterisk  had  more 


dissatisfied  than  satisfied  users.  The  source  of  this 
dissatisfaction  for  these  six  factors  was  expressed  in  the 
literature  reviewed  in  Chapter  3.  The  factor,  Time  required 
for  new  development  (F32),  was  cited  often  as  being 
excessive  from  the  user's  standpoint  (Neuschel,  1971;  Stone 
and  Tarnowieski,  1972;  Colton,  1973).  Similarly,  Fitts 
(1971)  found  unresponsiveness  to  the  Processing  of  change 
requests  (F2)  a major  irritant  to  users.  These  two  factors, 
in  conjunction  with  Flexibility  (F38)  and  Integration  (F39), 
seem  to  indicate  a frustration  on  the  part  of  the  user  to 
increasingly  bring  the  power  of  the  computer  to  bear  on  his 
problems . 


In  his  survey  of  616  computer  users,  Lucas  (1973a) 
found  that  the  desire  for  more  training  was  the  second  most 
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Table  5.11 


Number  of  Dissatisfied  Subjects 


Factor 

i 

Number  of 

Subj  ects 

Factor 

i 

Number  < 
Subject! 

1 

3 

21 

4 

2 

13* 

22 

8 

3 

7 

23 

8 

4 

9 

24 

6 

5 

10 

25 

1 

6 

2 

26 

4 

7 

5 

27 

4 

8 

7 

28 

6 

9 

6 

29 

3 

10 

7 

30 

7 

11 

8 

31 

6 

12 

9 

32 

20* 

13 

4 

33 

9 

14 

18* 

34 

1 

15 

12 

35 

13 

16 

10 

36 

7 

17 

4 

37 

6 

18 

6 

38 

15* 

19 

4 

39 

15* 

20 

14* 

*More  dissatisfied  than  satisfied  subjects 
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"extremely  important"  by  at  least  one  subject. 

The  order  of  the  factors  rated  most  important  overall 
also  had  logical  appeal.  The  top  rated  four  factors  seemed 
to  indicate  the  importance  of  computer  information  products 
and  services  that  were  accurate  2 8^  » reliable  (F^g) , 

timely  (Fg) , and  relevant  (F25)  to  the  user's  needs.  It 
appeared  intuitive  that  these  four  factors  formed  the  crux 
of  the  satisfaction  construct. 

The  factors  causing  dissatisfaction  were  also 
revealing.  The  number  of  subjects  expressing  a negative, 
hence  dissatisfied,  score  for  each  factor  is  shown  in  Table 
5.11.  The  factors  identified  by  an  asterisk  had  more 
dissatisfied  than  satisfied  users.  The  source  of  this 
dissatisfaction  for  these  six  factors  was  expressed  in  the 
literature  reviewed  in  Chapter  3.  The  factor,  Time  required 
for  new  development  (F32) , was  cited  often  as  being 
excessive  from  the  user's  standpoint  (Neuchel,  1971;  Stone 
and  Tarnowieski,  1972;  Colton,  1973).  Similarly,  Fitts 
(1971)  found  unresponsiveness  to  the  Processing  of  change 
requests  (F2)  a major  irritant  to  users.  These  two  factors, 
in  conjunction  with  Flexibility  (F3g)  and  Integration  (F39) , 
seem  to  indicate  a frustration  on  the  part  of  the  user  to 
increasingly  bring  the  power  of  the  computer  to  bear  on  his 
problems . 


In  his  survey  of  616  computer  users,  Lucas  (1973a) 
found  that  the  desire  for  more  training  was  the  second  most 
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frequent  comment  in  the  open-ended  section  of  the  question- 
naire.  If  the  desire  for  more  training  may  be  translated  as 
dissatisfaction,  a similar  finding  was  reflected  in  the 
large  number  of  subjects  dissatisfied  with  the  Degree  of 
training  (F^)  factor.  Likewise,  Top  Management  involvement 
(F2q)  was  consistently  characterized  as  vital  to  successful 
utilization  of  computer-based  information  systems  within  an 
organization  (McKinsey  and  Company,  1969;  Hofer,  1970; 

Fitts,  1971;  Colton,  1973).  It  may  be  inferred  that  lack  of 
such  involvement  could  lead  to  dissatisfaction;  however, 
there  was  no  evidence  to  collaborate  this  with  the  subject 
responses  of  dissatisfaction  for  this  factor. 

Although  the  specific  factors  cited  by  the  importance 
rankings  or  by  the  number  of  dissatisfied  subjects  have  no 
quantitative  relationship  to  the  expected  network  of  varia- 
bles within  the  satisfaction  construct,  their  appearance  and 
order  in  relation  to  the  other  factors  did  not  appear  illog- 
ical. This  was  an  important  consideration  for  construct 
validity  because  illogical  combinations  of  factors  could 
have  evolved.  If,  for  example,  the  factor  Accuracy  (F2s) 
had  achieved  an  extremely  low  importance  ranking,  either  the 
construct  definition  or  the  measurement  process  would  have 
been  subject  to  question.  Instead,  the  evidence  of 
construct  validity  for  the  user  satisfaction  measurement 
instrument  was  reinforced  by  the  rankings. 
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In  order  to  further  examine  the  subject  responses,  the 
39  factor  scores  for  each  of  the  29  subjects  were  subjected 
to  a factor  analysis  using  the  SPSS  factor  analysis  program. 

The  intent  was  to  determine  if  there  was  a logical  pattern 
among  the  39  factor  intercorrelations.  It  should  be  noted 
that  the  factor  analysis  program  will  always  generate 
factor-dimensions  based  on  the  empirical  input.  As  such, 
it  is  strictly  a tool  to  aid  in  the  analysis  of  the  data. 

Substantive  meaning  of  the  variables  which  load  highly  on 
the  factor-dimensions  is  strictly  a function  of  interpre- 
tation, not  of  the  loadings  themselves.  The  primary  use  of 
the  factor  analysis,  therefore,  is  to  aid  in  highlighting 
the  patterns  which  exist  and  to  stimulate  the  evaluation  of 
the  ramifications  of  the  patterns. 

Eight  factor-dimensions  were  extracted  from  the  meas- 
urement scores  to  "explain"  the  intercorrelations.  The 
factor  loadings  were  examined  to  find  those  variables  (user 
satisfaction  factors)  which  were  highly  loaded  (greater  than 
.50)  on  a single  factor-dimension.  Variables  that  were 
loaded  on  more  than  one  factor-dimension  were  ignored.  The 

I 

variable  clusters  and  the  factor  loadings  are  shown  in  Table 
5.12.  The  complete  factor-dimension  matrix  is  contained  in 
Appendix  5.5. 

The  clustering  shown  in  Table  5.12  was  only  suggestive 

\ 

of  a logical  underlying  structure.  Four  of  the  factors  (F^  ' 

Fig,  F30,  F37)  loading  highly  on  Dimension  1 were  associated 
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Table  5.12 

Factor  Analysis  Loadings 
Dimension  Loading  i Factor  (F^) 


1 

.8162 

1 

Relationship  with  EDP  staff 

.8105 

8 

Perceived  utility 

.6125 

11 

Expectations 

.6536 

18 

Attitude  of  the  EDP  staff 

.8599 

30 

Communications  with  EDP  staff 

.8512 

37 

Technical  competence  of  the  EDP 

staff 

2 

.8654 

19 

Reliability 

.7077 

21 

Format  of  output 

.6834 

27 

Job  effects 

.7996 

28 

Accuracy 

.7461 

29 

Precision 

.8464 

36 

Completeness 

3 

.7397 

4 

Organizational  competition  with 

EDP  staff 

. 74^8 

14 

Degree  of  training 

4 

.6795 

6 

Timeliness 

.7300 

17 

Currency 

.6443 

33 

Feeling  of  control 

5 

.7542 

26 

Volume  of  output 

6 

.6390 

2 

Processing  of  change  requests 

.7878 

20 

Top  management  involvement 

.5557 

32 

Time  required  for  new  development 

7 

.6493 

25 

Relevancy 

8 

.7184 

7 

Charge-back  method  of  payment 

.6539 

13 

Security  of  data 
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with  the  EDP  staff.  The  other  txra  factors  (Fg , F^)  apeared 
related  to  one  another  rather  than  to  the  other  four. 
Dimension  2 involved  factors  (F^g,  F21,  F28»  F29>  F36)  which 
are  all  associated  with  what  might  be  termed  the  "quality  of 
information  products".  The  other  factor  (F27)  is  not 
clearly  related  to  the  others.  Other  factor  pairs  which 
were  considered  to  have  a natural  affinity  were  clustered  on 
Dimension  4 (Fg,  F17)  and  Dimension  6 (F2,  F32) . The  other 
factors  included  in  Table  5.12  were  not  readily  explained. 

The  confirmation  of  construct  validity  was  less  conclu- 
sive than  either  content  or  predictive  validity;  however, 
substantial  evidence  was  presented  to  that  end.  It  was  felt 
that  total  confirmation  of  construct  validity  was  beyond  the 
scope  of  this  research;  yet,  the  development  of  the  user 
satisfaction  measurement  instrument  was  a necessary 
accomplishment  toward  that  objective. 

SUMMARY 

The  user  satisfaction  measurement  instrument  was 
evaluated  for  the  properties  of  reliability  and  validity. 
These  two  properties  were  considered  fundamental  to  the 
achievement  of  the  research  objective.  The  evaluation 
included  the  administration  of  the  instrument  to  two  groups 
of  subjects. 

Reliability  coefficients  were  calculated  for  each  of 
the  39  factors.  The  coefficients  reflected  the  proportion 
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of  total  variance  in  the  factor  scores  attributable  to 
measured  individual  differences.  The  high  reliability 
coefficients,  ranging  from  .7527  to  .9828,  indicated  that 
the  errors  of  measurement  were  insignificant.  This  evidence 
ratified  the  reliability  property  of  the  measurement 
instrument . 

Three  aspects  of  validity  were  examined.  The  support 
was  stronger  for  content  and  predictive  validity  than  for 
construct  validity;  however,  each  aspect  implied  the  other. 
Total  confirmation  of  validity  was  felt  to  be  a matter  of 
degree  rather  than  a dichotomy  of  "is  valid"  or  "is  not 
valid".  The  evidence  presented  in  this  evaluation  denoted 
all  of  the  aspects  of  validity  to  a substantial  degree. 
Further  confirmation  of  the  validity  of  the  user  satis- 
faction measurement  instrument  was  expected  to  result  from 
the  use  of  the  instrument  in  future  research. 
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Implementation  of  this  model  imposed  two  requirements. 
One  was  the  identification  of  the  factors  which  comprised 
the  domain  of  the  user  satisfaction  construct.  The  second 
was  the  measurement  of  the  user's  attitudinal  feelings 
toward  those  factors.  The  scaling  methodologies  commonly 
utilized  in  the  measurement  of  attitudes  were  evaluated,  and 
the  semantic  differential  technique  was  selected  as  the  most 
appropriate  for  measuring  user  satisfaction.  Semantic 
differential  scales  were  demonstrated  to  be  interval  scales 
which  anchor  a bipolar  continuum  with  the  zero  point  of 
attitude  intensity  located  at  the  midpoint  of  the  range  of 
the  scale  sums.  These  metric  properties  conformed  to  the 
model  requirements. 


The  factors  which  comprised  the  domain  of  the  user 
satisfaction  construct  were  tentatively  identified  through  a 
literature  review.  This  list  of  factors  and  their  opera- 
tional definitions  were  reviewed  and  supplemented  by  five 
independent  judges.  The  factors  identified  in  this  manner 
were  considered  to  be  the  "important"  factors  which  influ- 
ence a user's  satisfaction.  The  list  was  defined  to  be 


complete  if  the  probability  exceeded  .90  that  any  factor 
mentioned  by  a user  in  an  interview  was  contained  in  the 
list.  Thirty- two  middle  management  users  in  eight  different 
organizations  were  interviewed  to  elicit  the  factors  which 
they  considered  important.  A hypothesis  test  of  the  factors 
discussed  in  the  interviews  supported  the  completeness  of 
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the  factor  list  at  the  .01  level  of  significance.  These  39 
factors,  defined  in  Appendix  3.8,  constituted  the  user 
satisfaction  construct  domain. 

The  construction  of  the  measurement  instrument 
integrated  the  construct  domain  with  the  semantic  differ- 
ential methodology.  The  specific  evaluation  scales  for  each 
factor  in  the  domain  were  defined  by  bipolar  adjective 
pairs.  Four  adjective  pairs  (scales)  for  each  factor  were 
selected  in  a two-step  Delphi  approach  in  concert  with  five 
university  professors.  These  four  scales  were  coupled  with 
an  internal  consistency  scale  and  an  importance  weighting 
scale  to  operationalize  the  measurement  of  a user's  feeling 
toward  each  factor.  After  an  evaluation  of  the  decisions  of 
format,  the  scales  were  assembled  into  booklet  form  with 
appropriate  instructions  for  completion  to  create  the 
measurement  instrument. 

An  evaluation  of  the  measurement  instrument  commenced 
with  the  administration  of  the  instrument  to  the  32  users 
who  had  participated  in  the  initial  interviews.  Twenty-nine 
middle  managers  completed  and  returned  the  booklets.  Their 
responses  were  analyzed  both  statistically  and  subjectively 
to  substantiate  the  reliability  and  validity  properties  of 
the  instrument.  It  was  concluded  that  the  results  of  this 
evaluation  denoted  a successful  demonstration  of  the 
research  purpose,  the  measurement  of  user  satisfaction. 
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CONCLUSIONS 

The  general  conclusions  drawn  from  this  study  are  that 
the  research  objectives  were  achieved.  User  satisfaction 
can  be  defined  within  a domain  of  factors,  and  the  user's 
feelings  toward  those  factors  can  be  measured  with  the 
measurement  instrument  as  constructed.  The  specific 
conclusions  warranted  by  this  study  are  enumerated  and 
discussed  as  follows: 

1.  The  user  satisfaction  construct  was  adequately 
defined  by  the  39  factors  listed  in  Appendix  3.8.  Thirty- 
two  users  in  eight  organizations  were  interviewed.  During 
those  interviews,  each  factor  that  was  mentioned  was 
identified  as  a factor  important  to  that  user.  Multiple 
mentions  of  the  same  factor  in  a single  interview  were 
counted  as  only  one  mention.  The  total  number  of  factors 
mentioned  by  the  subjects  was  638,  an  average  of  19.9 
factors  per  subject.  Only  13  of  the  638  factor  mentions 
were  factors  not  identified  in  the  factor  list  shown  in 
Appendix  3.8.  This  supported  the  hypothesis,  at  the  .01 
level  of  significance,  that  the  probability  exceeded  .90 
that  a factor  considered  important  by  a user  was  included  in 
the  measurement  process.  The  hypothesis  implied  that  the 
preponderance  of  things  that  mattered  to  a user  were 
measured  by  the  instrument  as  defined. 

2.  Weighting  of  factor  scores  by  importance  ratings 
was  unnecessary.  The  correlation  coefficient  for  weighted 


and  unweighted  scores  was  .9968.  The  explanation  for  the 
high  correlation  was  revealed  in  a detailed  examination  of 
the  subject  responses.  Important  factors  were  typically 
rated  more  extremely  on  the  scales  than  less  important 
factors  which  were  rated  more  neutrally.  This  "internal 
weighting"  of  the  factors  minimized  the  effect  of  the 
weighting  function  ascribed  by  the  importance  scale.  This 
reduced  the  information  value  of  the  scale  to  the  simple 
identification  of  factors  considered  directly  important  by 
an  individual  subject. 

The  measurement  instrument  was  reliable.  An 


of  variance  approach  was  used  to  partition  the 
-ance  obtained  with  a set  of  measured  scores.  The 


reliability  coefficient  was  calculated  for  each  factor  as 
the  proportion  of  the  total  variance  attributable  to  "true 
variance".  The  coefficients,  which  ranged  from  .7527  to 
.9820,  indicated  that  the  variance  attributable  to 
measurement  error  was  negligible.  This  implied  that  the 
variance  in  the  set  of  scores  was  almost  wholly  accounted 
for  by  the  measurement  technique. 

4.  The  measurement  instrument  was  valid.  Three 


aspects  of  validity  were  examined.  Content  validity  was 
indicated  subjectively  by  the  definition  of  the  construct 
domain  and  the  careful  application  of  the  semantic  differ 
ential  methodology.  In  addition,  the  subject  response 
structures  exhibited  the  characteristics  of  homogeneity, 
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unidimensionality , and  scale  discrimination.  These 
characteristics  were  analytically  determined. 

Predictive  validity  was  verified  by  the  correlation 
between  the  subject  self-assessments  of  satisfaction  and 
their  measured  scores.  The  Pearson  product-moment  corre- 
lation coefficient  was  .7899,  significant  at  the  .001  level. 
The  linearity  exhibited  by  the  group  means  also  followed  the 
expected  pattern.  Another  "known  group"  criterion  was  less 
conclusive.  Subjects  identified  by  an  external  judge  only 
partially  followed  the  expected  scoring.  This  was  attri- 
buted to  erroneous  assignment  of  subjects  to  groups  and  was 
considered  less  significant  than  the  self-assessment 
results . 

Construct  validity  was  strongly  suggested  by  the 
characteristics  of  content  and  predictive  validity.  It  was 
further  confirmed  by  the  positive  importance  means  ascribed 
to  the  factors  by  the  subjects.  Furthermore,  the  specific 
factors  identified  as  the  most  important  by  the  subjects 
appeared  eminently  logical.  The  factors  which  were  rated 
dissatisfactory  by  a majority  of  the  subjects  agreed  with 
the  findings  suggested  in  other  studies.  A factor  analysis, 
however,  was  only  suggestive  of  an  expected  underlying 
structure  for  the  factor  intercorrelations. 


The  evidence  considered  in  entirety  confirmed  the 
validity  of  the  measurement  instrument.  This  implied  that 
the  subject  attribute  that  was  being  measured  reliably  was 
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the  attribute  (construct)  of  user  satisfaction. 

5.  The  measurement  instrument  was  objective.  The 
rules  for  administration  and  scoring  of  the  instrument  were 
completely  determined.  Moreover,  there  was  no  verbalization 
required  of  the  subject.  Therefore,  the  instrument  was 
amenable  to  cross-research  comparison  on  a common  basis. 

6.  The  measurement  metric  was  directly  interpretable. 
Each  factor  score  ranged  from  +3  to  -3  with  equidistant 
intervals  associated  with  fixed  responses  (for  example,  +2 
was  associated  with  "quite  satisfied") . Each  satisfaction 
score  ranged  from  +117  to  -117  with  the  positive  or  negative 
sign  indicating  satisfaction  or  dissatisfaction,  respec- 
tively. The  intensity  of  the  satisfaction  or  dissatis- 
faction was  reflected  by  the  magnitude  of  the  score.  This 
direct  interpretation  precluded  the  requirement  to  develop 
standard  sample  scores  for  comparison  purposes. 

7.  The  measurement  instrument  was  economical.  The 
user  time  to  complete  the  instrument  ranged  from  15  to  25 
minutes.  This  time  could  be  reduced  somewhat  by  the 
elimination  of  the  interval  consistency  and/or  importance 
scales  (depending  upon  the  measurement  objective).  The 
format  of  the  instrument  was  designed  to  facilitate 
independent,  unsupervised  completion  by  the  subjects. 

Scoring  of  the  responses  was  conducive  to  straightforward 
input  for  computer-assisted  analysis.  The  instrument  was 
subject  to  rapid  modification  by  the  addition/deletion  of 
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factors  and/or  scales.  Although  such  modifications  would 
lack  the  reliability  and  validity  assessments  of  this  study, 
thoughtful  changes  could  be  undertaken  with  minimal  risk. 

RECOMMENDATIONS 

The  results  and  conclusions  of  chis  study  indicate 
several  directions  for  additional  research.  The  specific 
recommendations  are  enumerated  and  discussed  as  follows: 

1.  The  validation  of  the  measurement  instrument  could 
be  extended.  Any  measurement  instrument,  especially  a new 
one,  benefits  by  multiple  studies  directed  at  val • dation. 

The  accumulated  evidence  fosters  refinement,  improvement, 
and  acceptance  of  the  instrument.  One  conceivable  approach 
to  this  type  of  research  could  be  the  identification  and 
evaluation  of  organizations  considered  to  have  effective  or 
ineffective  EDP  operations.  Administration  of  the  user 
satisfaction  instrument  to  users  in  these  organizations 
would  provide  a contrast  to  the  a priori  judgments.  The 
organizations  could  possibly  be  identified  from  the  major 
computer  vendors'  "showcase"  and  "problem"  installations. 

2.  Each  of  the  39  construct  factors  could  be 
extensively  examined.  This  examination  could  lead  to  the 
identification  of  the  structural  dimensions  and  charac- 
teristics of  each  factor  with  an  understanding  of  what  leads 
to  satisfaction.  Such  information  would  be  useful  in  the 
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development  of  a performance  plan  for  the  EDP  organizational 
unit . 

3.  The  measurement  instrument  could  be  used  to 
contrast  user  satisfaction  structures  with  organizational 
structures  (both  EDP  and  corporate)  and  policy  structures 
within  the  organizations.  An  understanding  of  the  organi- 
zational and  policy  conditions  which  lead  to  greater  user 
satisfaction  could  lead  to  better  decisions  in  establishing 
those  conditions. 

4.  The  use  of  the  measurement  instrument  could  be 
integrated  into  an  overall  productivity  model  which  includes 
both  effectiveness  and  efficiency  measures  for  the  EDP  unit. 
This  would  allow  the  setting  of  target  goals  for  the  EDP 
staff  and  the  measurement  of  progress  toward  those  goals. 

5.  The  measurement  instrument  could  be  used  to  explore 
the  dynamics  of  user  satisfaction  over  time  and  circum- 
stance. Thus,  the  effects  of  technological  and  other 
changes  could  be  catalogued  and  evaluated.  Some  of  the 
marginal  effects  that  could  be  examined  include  distributed 
versus  centralized  processing,  on-line  versus  batch  customer 
support,  and  charge-back  algorithms  versus  fixed  allocation 
of  costs.  This  research  would  be  amenable  to  large  sample 
studies  in  which  other  variables  in  addition  to  user  satis- 
faction are  collected  by  subject. 

6.  The  measurement  instrument  could  be  used  to 
establish  comparative  standards  of  user  satisfaction  that 
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are  appropriate  for  varying  circumstances.  Although  the 
user  satisfaction  metric  is  directly  interpretable, 
standards  would  be  useful  for  management  comparisons.  For 
example,  if  the  average  user  satisfaction  score  in  an 
organization  is  40,  the  users  may  be  classified  as  "slightly 
satisfied";  however,  this  "slightly  satisfied"  rating  may  be 
extremely  low  or  high  when  compared  to  the  average  user 
scores  of  organizations  of  similar  type  and  size,  EDP 
budget,  etc.  The  identification  of  appropriate  categories 
and  category  stratifications  would  facilitate  the  estab- 
lishment of  such  standards. 

7.  The  measurement  instrument  could  be  adopted  for  any 
research  requiring  the  quantification  of  the  user  satis- 
faction variable.  The  absence  of  any  comparable  instrument 
coupled  *7ith  a growing  interest  in  this  research  area  offers 
an  excellent  opportunity.  Many  advantages  would  accrue  by 
the  widespread  adoption  of  a single  basis  of  research 
measurement.  Researchers  would  not  have  to  develop  their 
own  instruments,  and  cross-research  comparison  would  be 
greatly  facilitated.  Such  acceptance  of  the  instrument 
could  lead  to  a much  richer  and  rapid  assimulation  of 
knowledge.  That  is  ultimately  one  of  the  goals  of  all 
research. 
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Appendix  3.2 


CORRESPONDENCE  WITH  SUBJECTS 


The  following  letters  and  enclosures  were  used  to 
solicit  each  subject's  participation  in  three  phases  of  the 
research:  initial  interview,  factor  ranking,  and 

questionnaire  completion.  The  addresses  are  deliberately 
omitted.  The  enclosure  for  the  first  letter  was  a self- 
addressed  postcard  to  allow  the  subject  to  decline  to 
participate.  The  enclosures  for  the  second  letter  were  a 
list  of  factors  specified  in  the  subject's  interview  and  the 
definitions  for  that  specific  set  of  factors.  A sample 
factor  list  is  shown;  the  factor  definitions  are  omitted. 

The  enclosure  for  the  third  letter  was  the  measurement 
instrument  shown  in  Appendix  4.1,  and  it  is  not  repeated 
here.  The  letters  have  been  reduced  photographically. 
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ARIZONA  STATI: 

I ' N I V I:  R S I T V 

SCHOOL  Of  ENGINEERING 

Industrial  Engineering 


.TIVIIT.  XRI/flN  X .v*.>l 


February  20,  1976 


I an  an  Arizona  State  University  student  curre'tly  working  toward  a Fh.D. 
degree  in  Industrie’  Engineering.  I rave  i'itiated  a research  project  to 
discover  and  to  measure  the  factors  whi  cr  cause  e user  to  te  satisfies  or 
dissatisfied  with  the  computer-based  in'crrscic"  systems  arc  services  orovided 
within  an  organization.  The  achievement  o*  this  '-search  coal  will  result  in 
the  improvement  of  the  systems  provided  rcr  the  users. 

Your  present  oositior.  indicates  that  you  cc.ld  -eke  e va’x.d’cle  contribution 
to  this  research  effort.  I would  greatly  egp'ec’aoe  ..cur  participation.  Your 
involvement ’./ill  consist  of  an  interview  to  discuss  ytur  experiences  with 
computer-based  information  services  arc  two  fs". 'c.-.-.c  questionnaires.  The  tire 
required  is  minimal.  in  contrast  to  the  tfefits  whicn  ray  result. 


Although  no  adverse  effects  are  ■fpre;ee" 


K.n-  r*  ,/n 


our  D5r 


participation. 


a project  identification  number  will  be  -sec  t:  prctect  the  ancry--' ty  of  the 
data  collected.  The  link  between  the  identity  s'  you  end  your  organization  and 
the  assoc'ated  number  will  be  known  only  to  -_.se  if  arc  ry  c;-~iccee  chair-an, 

Dr.  James  E.  Bailey.  f 

I hope  that  you  will  ecree  to  join  in  c-ts  resea-ch  project.  I wifi 
contact  you  via  tele  shore  to  arrange  an  a:  rent  -n’ess  you  -efurc,  the  enclosed 

ca^d  declining  the  request.  I will  glad..  ens'.=r  an.  ejections  at  that  time. 


Thank  you  fo-'  you r cooperation. 


am  leer,  mg  ror.uaro  to  meeting 
Sincerely, 

Surry  Fearson 


I 
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ARIZONA  trTATl: 

l1  N 1 V I;  R O'  I I V Tivn  vm/ir.  v - :-i 

SCHOOl  OF  ENGINEERING 

Industrial  Engineering 


I want  to  convey  my  appreciation  for  your  participation  the  recent 
interview  to  support  my  research.  The  information  that  you  provided  is  a 
valuable  contribution  toward  the  identification  cf  'actors  ..r.ich  cause  a user 
to  be  satisfied  or  dissatisfied  with  the  computer-based  information  systems  and 
services  provided  within  an  organization.  I have  reviewed  the  tape  recordino 
of  the  interview  and  have  extracted  the  factors  that  ypu  discussed  at  that  tire. 

These  factors  are  listed  on  the  second  pace.  ! would  now  like  for  you  to 
rank  these  factors  in  their  order  cf  importance  as  i--" ue-ces  or.  your  o.-.n  satis- 
faction. A rank  of  ore  should  be  assisted  ta  tire  fact:'  drat  vou  consider  tne 
most  important  influence,  rank  two  tc  tie  sect  os  rest  importers,  -ptp.  Assign 
a unique  rank  tc  each  factor.  Operational  definitions  fo-  each  factor  are 
provided  for  clari f ication  of  the  terrinolooy.  Please  return  tne  marked  factors 
to  me  via  the  enclosed  envelope. 

Your  cooperation  in  this  project  nas  made  my  dash  not  only  easy  but  also 
very  pleasant.  In  that  same  spirit,  I request  that  vou  rasot'd  pmcrptlv  with 
the  rankings  so  that  the  subsequent  research  nhesus  can  proceed  apace.  Thank 
you. 


S i nepr1?]  v , 


Sammy  Pearson 

Enclosures 


dr 


I 


I 


. 


y 
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Assign  ranks  1 through  22  to  each  factor  listed  below.  Rank  1 is  assigned 
to  the  most  important  factor.  Rank  22  is  assigned  to  the  least  important 
factor.  Each  factor  must  be  assigned  a unique  rank. 


Ranking  Factor 

3 Accuracy 

JR.  Format  of  output 

^ Reliability 

5(2.  Processing  of  change  requests 

Confidence  in  the  systems 
7 Job  effects 

3i  Perceived  utility 

UL.  Feeling  of  participation 

Understanding  of  systems 
Technical  competence  of  the  EDP  staff 
Relevancy 

Degree  of  training 
Communication  with  the  EDP  staff 
^ Documentation 

J6  Feeling  of  control 

—Let Error  recovery 

Expectations 

3.  / Mode  of  interface 

aa  Priorities  determination 

/ 7 Relationship  with  the  EDP  staff 

-IP. . Rate  of  change 

_ ML - Standardization  of  systems 


R. 


JRL 


A 


RETURN  THIS  PAGE  VIA  THE  ENCLOSED  ENVELOPE 


Appendix  3.3 

BACKGROUND  QUESTIONNAIRE  AND  CONSENT  STATEMENT 

The  following  questionnaire  and  consent  statement  were 
completed  by  each  subject  prior  to  the  interview.  The 
satisfaction  scale  shown  on  page  two  was  on  the  reverse  side 
of  the  questionnaire  and  was  completed  at  the  conclusion  of 
the  interview. 
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Name 

Office  Address 

Home  Address 


Project  Number  

Phone 


Phone 


r 


» 


Send  research  correspondence  to  ( 

( 


) office 
) home 


My  functional  area  is:  (check  all  that  apply) 


( ) Accounting 
( ) Administration 
( ) Finance 


( ) Marketing 
( ) Operations 
( ) Personnel 


( ) Planning/Budqet 
( ) Other  


I have  ( ) 0-3 
( ) 4-6 
( ) over  6 


years  experience  using  computer-based 
information  products  and  services. 


I use  ( ) hard-copy  output 

( ) terminals  (Check  all  that  apply) 

( ) computer-output  microforms 


I consider  myself  a ( ) heavy 

( ) medium 
( ) light 


user  of  computer-based  information 
products  and  services. 


I hereby  authorize  this  interview  to  be  tape  recorded.  My  participation 
is  voluntary,  and  I understand  that  I may  freely  withdraw  such  participation 
at  any  time. 


Date 


CONFIDENTIAL 
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fSCT  iV-  . . J.  I'-* 


INTERVIEW  GUIDELINES 


| RECORDER  OFF  | 
**•*  *** 


-»*  **■****•*■#••(!•'«■*■«•*•  »•»*  *■•»**•«•##*  •*•!!•  *#■*•(!■  **■«•*#  -it  #**•*■  *#  -K-  **•***■»(■•?{•*  * 


Mr  / Ms  _j" , 

This  interview  is  being  conducted  as  part  of  a research 
project  to  discover  and  to  measure  the  factors  leading  to 
satisfaction  or  dissatisfaction  with  the  computer-based 
information  systems  and  services  provided  within  an 
organization.  As  the  cover  letter  explained,  this  inter- 
view is  the  first  of  several  actions  in  which  you  will  be 
asked  to  participate.  You  were  selected  because  you  are  in 
a position  to  provide  meaningful  information  about  the  re- 
search questions.  The  anonymity  of  the  data  collected  from 
you  will  be  protected  by  the  methods  described  in  the  cover 
letter.  Do  you  have  any  questions? 


**  ***  *•  * * w***-******  # -is-  -ss-  ****  * * *■>:-■**■*•  *•  •>  * -»■*•**■  -:>■  **#***••»• 


Before  we  begin,  I would  like  for  you  to  fill  out  this 
brief  background  form.  The  information  will  be  used  to 
insure  that  the  sample  distribution  is  adequate,  I would 
also  like  for  you  to  read  and  to  sign  the  consent  form  at 
the  bottom  of  the  page.  This  sinply  indicates  your  willing- 
ness to  participate  freely  and  gives  me  permission  to  record 


our  conversations. 


«•»•*#  #•#•*•#•  a-***##  #*■*#■*•»****#**•*■■»*****  **###*■**■#•**•«•*****• 


* FILL  OUT  BACKGROUND  FORM  / CONSENT  FORM  £ 


| RECORDER  ON  | 

#*•**•**•  **•*"!:-  -ft  #««##*#«####»  *•&**•*•»#•*■!<■■»*•«•#-»*  **■**#  **•*•  ########«*« 

Do  you  have  any  objections  to  recording  this  interview? 


Nov/,'  I.  would  like  for  you  to  think  of  a time  when  you 
felt  especially  satisfied  or  dissatisfied  with  the  computer- 
based  information  services  or  products  that  you  were 
receiving.  This  includes  any  time  during  your  working 
career.  Please,  describe  that  situation  for  me. 

PROBES 

1.  7/hat  did  these  events  mean  to  you? 

2.  Can  you  tell  me  more  precisely  why  you  felt  the  way 
you  did  at  the  time? 

3.  Ho v/  did  it  specifically  affect  you? 

4.  Can  you  give  me  an  example  of  how  it  affected  your  job? 

5.  How  long  did  it  last? 

6.  Did  anything  occur  that  changed  the  situation  and  your 
feelings? 

7.  Why  did  that  (satisfy/dissatisfy)  you? 

8 . How  do  you  mean? 

9.  If  that  happened  today,  would  it  have  the  same  effect? 

10.  Was  there  anything  during  that  period  that  (displeased/ 
pleased)  you? 

■*■*•*•*  * * # #■*#****■&*****#*■«■***•■»■*  #*■*  #*•*■*#•*•&*•&****  -if  * ■»*•»•  ■»■»■•*  #***■!{■  # 

Now  that  you  have  described  a time  when  you  felt 

especially  with  your  computer-based  support, 

will  you  describe  a time  when  you  were  especially  . 
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How  do  you  feel  about  the  computer-based  information 
support  you  currently  receive? 

-K-  -SHt  ****-******•*■!(••*•*•**■*•&■»■•&■>!•**  *■»*•«•  tt*#- ■&**•*■*!•*  * 

*■&# 

I PROBE  | 

#**»***#■*#*'#*##**#*##**#•&****•*#****##***»***#•»*#*****«•#**#■ 

Considering  everything  with  regard  to  the  computer 
support  you  now  receive,  where  on  this  satisfaction  scale 
would  you  consider  yourself? 

******#*##**##***■*•**#•*•*  ■S-*##-****-*#-**************###**#*#.*.#*. 

***• 

| MARK  SATISFACTION  SCALE  f 

*****#■*■»**■*•*■«•**•»*•&•*•&■*•»•«•**«***■!!■•**•*•*#■*•*•*•■»■*  #****•*****■#••»•**»*** 

*■** 

| RECORDER  OFF  1 
$•**  ***• 

********************************************************** 

Thank  you  very  much  for  this  interview.  You  will 
receive  a follow-up  on  the  results  within  two  weeks.  At 
that  time  you  will  be  asked  to  rank  the  factors  which  you 
have  talked  about  in  this  interview  in  terms  of  importance. 
I would  appreciate  it  very  much  if  you  would  respond - 
immediately  when  you  receive  it  to  help  conclude  the  re- 
search within  my  time  limits.  It  should  take  no  more  than 
10  minutes  to  respond.  I appreciate  your  cooperation. 
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DATA  EXTRACTION  SYLLABUS 
Part  I i Purpose 

This  syllabus  provides  definitions  and  procedures 
for  extracting  data  from  tape  recordings  of  personal 
interviews.  The  data  of  interest  are  those  factors  which 
influence  a respondent  to  be  satisfied  or  dissatisfied 
with  the  computer-based  information  products  and  services 
provided  within  his  organization.  The  objective  of  this 
data  extraction  procedure  is  the  development  of  a list 
of  factors  for  each  individual  respondent  that  is  based 
solely  on  the  interview  statements. 


Part  II  i General  Definitions 
Factor:  Any  identifiable  element  which  influences  or 
contributes  to  a user's  satisfaction  or  dissatisfaction 
with  the  computer-based  information  support  that  is  pro- 
vided within  an  organization. 

Thought  Unit:  A statement  about  a single  event  or  con- 
dition that  led  to  a feeling  of  satisfaction  or  dissatis- 
faction, a single  characterization  of  a feeling,  or  a 
description  of  a single  event. 

EDPi  Electronic  Data  Processing. 
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PART  III:  FACTOR  DEFINITIONS 


Top  management  involvement:  The  positive  or  negative  degree  of  interest, 
enthusiasm,  support,  or  participation  of  any  management  level  above  the 
respondent's  own  level  toward  computer-based  information  systems  or 
services  or  toward  the  computer  staff  which  supports  them. 


Organizational  competition  with  the  EDP  unit:  The  contention  between  the 
respondent's  organizational  unit  and  the  EDP  unit  when  vying  for  organ- 
izational resources  or  for  responsibility  for  success  or  failure  of 
computer-based  information  systems  or  services  of  interest  to  iboth  parties. 


Organizational  Position  of  EDP  function:  The  hierarchical  relationship  of 
the  EDP  function  to  the  overall  organizational  structure. 

Priorities  determination:  Policies  and  procedures  which  establish  precedence 
for  the  allocation  of  EDP  resources  and  services  between  different 
organizational  units  and  their  requests. 

Charge-back  method  of  payment  for  services:  The  schedule  of  charges  and  the 
procedures  for  assessing  users  on  a pro  rate  basis  for  the  EDP  resources 
and  services  that  they  utilize. 

Relationship  with  the  EDP  staff:  The  manner  and  methods  of  interaction, 
conduct,  and  associ ation  between  the  user  and  EDP  staff. 

Communication  with  the  EDP  staff:  The  manner  and  methods  of  information 
exchange  between  the  user  and  the  EDP  staff. 

Technical  Competence  of  the  EDP  staff:  The  computer  technology  skills  and 
expertise  exhibited  by  the  EDP  staff. 

Attitude  of  the  EDP  staff:  The  willingness  and  commitment  of  the  EDP  staff 
to  subjugate  external , professional  goals  in  favor  of  organizationally 
directed  goals  and  tasks. 

Schedule  of  products  and  services:  The  EDP  center  time-table  for  production 
of  information  system  outputs  and  for  porvision  of  computer-based  services. 

Time  required  for  new  development:  The  elapsed  time  between  the  user's 

request  for  new  applications  and  the  design,  development,  and/or  imple- 
mentation of  the  application  systems  by  the  EDP  staff. 

Processing  of  change  requests:  The  manner,  method,  and  required  time  with 
whichthe  EDP  staff  responds  to  user  requests  for  changes  in  existing 
computer-based  information  systems  or  services. 

Vendor  support:  The  type  and  quality  of  the  service  rendered  by  a vendor, 
either  di rectly  or  indirectly,  to  the  user  to  maintain  the  hardware  or 
software  required  by  that  user.  A vendor  is  distinguished  by  his  external 
organizational  status. 
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Response/turnaround  titre:  The  elapsed  time  between  a user-initiated  request 
for  service  or  action  and  a reply  to  that  request.  Response  time  generally 
refers  to  the  elapsed  time  for  a terminal  type  request  or  entry.  Turnaround 
time  generally  refers  to  the  elapsed  time  for  execution  of  a program  sub- 
mitted or  requested  by  a user  and  the  return  of  the  output  to  that  user. 

Mode  of  interface:  The  method  and  medium  by  which  a user  inputs  data  to  and 
receives  output  from  the  EDP  center. 

Convenience  of  access:  The  ease  or  difficulty  with  which  the  user  may  act  to 
utilize  the  capability  of  the  computer  systems. 

Quality  of  systems:  The  degree  of  excellence  of  the  computer-based  information 
systems  as  manifested  in  the  six  attributes  of  the  system  outputs  - 
accuracy,  timeliness,  precision,  reliability,  currency,  and  completeness. 

Accuracy:  The  correctness  of  the  output  information. 

Timeliness:  The  availability  of  the  output  information  at  a time  suitable  for 
its  use. 

Precision:  The  variability  of  the  output  information  from  that  which  it 
purports  to  measure. 

Rel i ability:  The  consistency  and  dependability  of  the  output  information. 

Currency:  The  age  of  the  output  information. 

Completeness:  The  comprehensiveness  of  the  output  information  content. 

Flexibility:  The  capacity  of  the  information  system  to  change  or  to  adjust 
in  response  to  new  conditions,  demands,  or  circumstances. 

Format  of  output:  The  material  design  of  the  layout  and  display  of  the 
output  contents. 

Language:  The  set  of  vacabulary,  syntax,  and  grammatical  rules  used  to 
interact  with  the  computer  systems. 

Volume  of  output:  The  amount  of  information  conveyed  to  a user  from  computer- 
based  systems.  This  is  expressed  by  not  only  the  number  of  reports  or  out- 
puts but  also  by  the  voluminousness  of  the  output  contents. 

Relevancy:  The  degree  of  congruence  between  what  the  user  wants  or  requires 
and  what  is  provided  by  the  information  products  and  services. 

Error  recovery:  The  methods  and  policies  governing  correction  and  rerun  of 
system  outputs  that  are  incorrect. 
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Security  of  data:  The  safeguarding  of  data  from  misappropriation  or  un- 
authorized  alteration  or  loss. 

Documentation:  The  recorded  description  of  an  information  system.  This 
includes  formal  instructions  for  the  utilization  of  the  system. 

Expectations:  The  set  of  attributes  or  features  of  the  computer-based 

information  products  or  services  that  a user  considers  reasonable  and  due 
from  the  computer-based  information  support  rendered  within  his 
organization. 

Understanding  of  systems:  The  degree  of  comprehension  that  a user  possesses 
about  the  computer-based  information  systems  or  services  that  are  provided 
to  him. 

Perceived  utility:  The  user's  judgment  about  the  relative  balance  between 
the  cost  and  the  considered  usefulness  of  the  computer-based  information 
products  or  services  that  are  provided.  The  costs  include  any  costs 
related  to  providing  the  resource,  including  money,  time,  manpower,  and 
opportunity.  The  usefulness  includes  any  benefits  that  the  user  believes 
that  he  derives  from  the  support. 

Confidence  in  the  systems:  The  user's  feelings  of  assurance  or  certainty 
about  the  systems  ’jiwi  ch  support  him. 

Feeling  of  participation:  The  degree  of  involvement  and  commitment  which  the 
user  shares  with  the  EDP  staff  and  others  toward  the  functioning  of  the 
computer-based  information  systems  and  services. 

Degree  of  training:  The  amount  of  specialized  instruction  and  practice  that 
is  afforded  to  the  user  to  increase  his  proficiency  in  utilizing  the 
computer  capability  that  is  available  to  him. 

Job  effects:  The  changes  in  job  freedom  and  job  performance  that  are  ascer- 
tained by  the  user  as  resulting  from  modifications  induced  by  the  computer- 
based  information  systems  and  services. 

Feeling  of  control : The  user's  awareness  of  his  power  or  lack  of  power  to 

regulate,  direct  or  dominate  the  development,  alteration,  and/or  execution 
of  the  computer-based  information  systems  or  services  which  serve  him  in 
his  perceived  function. 
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Part  IV  » Procedure 

1.  Listen  to  the  entire  tape  and  concentrate  on  recog- 
nizing thought  units  as  they  are  expressed  by  the  res- 
pondent. 

2.  Rewind  the  tape. 

3.  Again,  listen  to  the  tape,  but  stop  anytime  that  a 
thought  unit  is  expressed. 

4.  If  the  factor  expressed  by  the  thought  unit  is  clear, 
write  down  the  factor  directly.  If  the  factor  is  not  clear, 
write  down  the  thought  unit  verbatim. 

5.  Evaluate  the  thought  unit  before  proceeding  with  the 
tape.  However,  a short  portion  of  the  interview  beyond 
the  thought  unit  in  question  may  clarify  it.  Decide  on 

a factor  that  the  thought  unit  expresses  and  write  it  down, 

6.  Proceed  with  the  tape  and  repeat  steps  3-5  until  the 
interview  portion  is  completed. 

7.  Divide  the  list  of  factors  that  were  extracted  from 
the  tape  into  two  groups.  Those  factors  that  were  defined 
in  Part  III  of  this  syllabus  are  listed  in  the  first  group 
using  the  nomenclature  as  defined.  The  second  group  of  fac- 
tors contain  those  factors  not  defined  in  this  syllabus. 

The  nomenclature  of  these  factors  must  be  devised  to  convey 
understanding  about  the  undefined  factors.  Duplicate  factors 
are  listed  only  once  in  the  final  two  groups. 

8.  This  completes  the  data  extraction. 
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FACTORS  MENTIONED  BY  SUBJECT 


An  "X"  in  the  following  matrix  identifies  the  specific 
factors  mentioned  by  a subject  during  the  subject's  inter- 
view. 
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FACTOR  RANKINGS  BY  SUBJECT 

t The  factors  mentioned  during  a subject's  interview  were 


ranked  in  order  of  importance.  The  most  important  factor 
was  assigned  the  value  one.  The  following  matrix  presents 
t the  factor  rankings  by  subject.  Subject  B3  declined  to  rank 

the  factors. 
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Appendix  3.8 

FACTOR  LISTING  AND  DEFINITIONS 

The  following  fold-out  list  of  factors  is  provided  for 
the  reader's  convenience.  The  list  is  supplemented  by  the 
factor  definitions. 


i 


FACTORS 


Relationship  with  the  EDP  staff 
Processing  of  change  requests 
Mode  of  interface 

Organizational  competition  with  EDP 
unit 

Confidence  in  systems 
Timeliness 

Charge-back  method  of  payment  for 
services 

Perceived  utility 
Vendor  support 
Language 
Expectations 
Error  recovery 
Security  of  data 
Degree  of  training 
Understanding  of  systems 
Feeling  of  participation 
Currency 

Attitude  of  the  EDP  staff 
Reliability 

Top  management  involvement 

Format  of  output 

Response/turnaround  time 

Priorities  determination 

Convenience  of  access 

Relevancy 

Volume  of  output 

Job  effects 

Accuracy 

Precision 

Communication  with  the  EDP  staff 
Organizational  position  of  the  EDP 
function 

Time  required  for  new  development 

Feeling  of  control 

Schedule  of  products  and  services 

Documentation 

Completeness 

Technical  competence  of  the  EDP 
staff 

Flexibility 
Integration  of  systems 
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Top  management  involvement:  The  positive  or  negative  degree 
of  interest,  enthusiasm,  support,  or  participation  of  any 
management  level  above  the  user's  own  level  toward  computer- 
based  information  systems  or  services  or  toward  the  computer 
staff  which  supports  them. 

Organizational  competition  with  the  EDP  unit : The 
contention  between  the  respondent's  organizational  unit  and 
the  EDP  unit  when  vying  for  organizational  resources  or  for 
responsiblity  for  success  or  failure  of  computer-based 
information  systems  or  services  of  interest  to  both  parties. 

Priorities  determination:  Policies  and  procedures  which 
establish  precedence  for  the  allocation  of  EDP  resources  and 
services  between  different  organizational  units  and  their 
requests . 

Charge-back  method  of  payment  for  services:  The  schedule  of 
charges  and  the  procedures  for  assessing  users  on  a pro  rata 
basis  for  the  EDP  resources  and  services  that  they  utilize. 

Relationship  with  the  EDP  staff:  The  manner  and  methods  of 
interaction,  conduct,  and  association  between  the  user  and 
the  EDP  staff. 

Communication  with  the  EDP  staff:  The  manner  and  methods  of 
information  exchange  between  the  user  and  the  EDP  staff. 

Technical  competence  of  the  EDP  staff:  The  computer 
technology  skills  and  expertise  exhibited  by  the  EDP  staff. 

Attitude  of  the  EDP  staff:  The  willingness  and  commitment 
of  the  EDP  staff  to  subjugate  external,  professional  goals 
in  favor  of  organizationally  directed  goals  and  tasks. 

Schedule  of  products  and  services : The  EDP  center  time- 
table  for  production  of  information  system  outputs  and  for 
provision  of  computer-based  services. 

Time  required  for  new  development : The  elapsed  time  between 
the  user's  request  for  new  applications  and  the  design, 
development,  and/or  implementation  of  the  applications 
systems  by  the  EDP  staff. 

Processing  of  change  requests:  The  manner,  method,  and 
required  time  with  which  the  EDP  staff  responds  to  user 
requests  for  changes  in  existing  computer-based  information 
systems  or  services. 

Vendor  support:  The  type  and  quality  of  the  service 
rendered  Dy  a vendor,  either  directly  or  indirectly,  to  the 
user  to  maintain  the  hardware  or  software  required  by  that 
user.  A vendor  is  distinguished  by  his  external 
organizational  status. 
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Response/ turnaround  time:  The  elapsed  time  between  a user- 
initiated  request  for  service  or  action  and  a reply  to  that 
request.  Response  time  generally  refers  to  the  elapsed  time 
for  a terminal  type  request  or  entry.  Turnaround  time 
generally  refers  to  the  elapsed  time  for  execution  of  a 
program  submitted  or  requested  by  a user  and  the  return  of 
the  output  to  that  user. 

Mode  of  interface:  The  method  and  medium  by  which  a user 
inputs  data  to  and  receives  output  from  the  EDP  center. 

Convenience  of  access:  The  ease  or  diffficulty  with  which 
the  user  may  act  to  utilize  the  capability  of  the  computer 
systems . 

Accuracy : The  correctness  of  the  output  information. 

Timeliness : The  availability  of  the  output  information  at  a 

time  suitable  for  its  use. 

Precision : The  variability  of  the  output  information  from 
that  which  it  purports  to  measure. 

Reliability:  The  consistency  and  dependability  of  the 
output  information. 

Currency : The  age  of  the  output  information. 

Completeness : The  comprehensiveness  of  the  output 
information  content . 

Format  of  output:  The  material  design  of  the  layout  and 
display  of  the  output  contents . 

Language : The  set  of  vocabulary,  syntax,  and  grammatical 

rules  used  to  interact  with  the  computer  systems. 

Volume  of  output:  The  amount  of  information  conveyed  to  a 
user  from  computer-based  systems.  This  is  expressed  by  not 
only  the  number  of  reports  or  outputs  but  also  by  the 
voluminousness  of  the  output  contents. 

Relevancy:  The  degree  of  congruence  between  what  the  user 
wants  or  requires  and  what  is  provided  by  the  information 
products  and  services. 

Error  recovery:  The  methods  and  policies  governing 
correction  and  rerun  of  system  outputs  that  are  incorrect. 

Security  of  data:  The  safeguarding  of  data  from  mis- 
appropriation or  unauthorized  alteration  or  loss. 
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Documentation : The  recorded  description  of  an  information 
system.  This  includes  formal  instructions  for  the 
utilization  of  the  system. 

Expectations : The  set  of  attributes  or  features  of  the 
computer-based  information  products  or  services  that  a user 
considers  reasonable  and  due  from  the  computer-based 
information  support  rendered  within  his  organization. 

Understanding  of  systems:  The  degree  of  comprehension  that 
a user  possesses  about  the  computer-based  informations 
systems  or  services  that  are  provided. 

Perceived  utility:  The  user's  judgment  about  the  relative 
balance  between  the  cost  and  the  considered  usefulness  of 
the  computer-based  informaton  products  or  services  that  are 
provided.  The  costs  include  any  costs  related  to  providing 
the  resource,  including  money,  time,  manpower,  and 
opportunity.  The  usefulness  includes  any  benefits  that  the 
user  believes  to  be  derived  from  the  support. 

Confidence  in  the  systems:  The  user's  feelings  of  assurance 
or  certainty  about  the  systems  provided. 

Feeling  of  participation:  The  degree  of  involvement  and 
commitment  which  the  user  shares  with  the  EDP  staff  and 
others  toward  the  functioning  of  the  computer-based 
information  systems  and  services . 

Feeling  of  control:  The  user's  awareness  of  the  personal 
power  or  lack  of  power  to  regulate,  direct  or  dominate  the 
development,  alteration,  and/or  execution  of  the  computer- 
based  information  systems  or  services  which  serve  the  user's 
perceived  function. 

Degree  of  training:  The  amount  of  specialized  instruction 
ana  practice  that  is  afforded  to  the  user  to  increase  the 
user's  proficiency  in  utilizing  the  computer  capability  that 
is  available. 

Job  effects:  The  changes  in  job  freedom  and  job  performance 
that  are  ascertained  by  the  user  as  resulting  from 
modifications  induced  by  the  computer-based  information 
systems  and  services. 

Organizational  Position  of  the  EDP  Function:  The 
hierarchical  relationship  of  the  EDP  function  to  the  overall 
organizational  structure. 


Flexibility  of  Systems : The  capacity  of  the  information 
system  to  change  or  to  adjust  in  response  to  new  conditions, 
demands,  or  circumstances. 


Appendix  4.1 

USER  SATISFACTION  MEASUREMENT  INSTRUMENT 

' 

The  following  pages  are  reproductions  of  the  pages  of 
the  user  satisfaction  measurement  instrument.  The  actual 
instrument  is  in  booklet  form. 

» 


To  Research  Participants: 

The  purpose  of  this  study  is  to  measure  how  you  feel  about 
certain  aspects  of  the  computer-based  information  products  and 
services  that  are  provided  to  you  in  your  present  position. 

On  the  following  pages  you  will  find  different  factors, 
each  related  to  some  aspect  of  your  computer-based  support. 

You  are  to  rate  each  factor  on  the  descriptive  scales  that 
follow  it,  based  on  your  evaluation  of  the  factor. 

A separate  scale  is  provided  for  you  to  express  how 
tmpontant  on  unimpontant  each  factor  is  to  you. 

The  scale  positions  are  defined  as  follows: 

term  X : : : : : : : : term  Y 

(1)  (2)  (3)  (4)  (5)  (6)  (7) 

(1)  zxtnmzly  X (5)  Atiglitly  Y 

(2)  quttz  X (6)  qultz  Y 

(3)  slightly  X (7)  extnzmzly  Y 

(4)  niuthzn  X non  'i\zqwilly  X on  Y ;dou  not  apply 

INSTRUCTIONS 

1.  Check  each  scale  in  the  position  that  describes  youn 
evaluation  of  the  factor  being  judged. 

2.  Check  evzty  scale,  do  not  omit  any.  N 

3.  Check  only  one  position  for  each  scale. 

4.  Check  in  the  space,  not  between  spaces.  : X : X : 

5.  Work  rapidly.  Rely  on  your  first  impressions. 

The  final  page  is  provided  for  any  comments  that  you  wish 
to  make.  Thank  you  very  much  for  your  cooperation.  Return 
this  booklet  via  the  enclosed  envelope. 

Sincerely, 
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1 Relationship  with  the  EDP*  staff 


* harmonious  : : : 

: : : dissonant 

good  : : : 

: : : bad 

cooperative  : : : 

: : : uncooperative 

candid  : : : 

: : : deceitful 

satisfactory  : : : 

: : : unsatisfactory 

To  me,  this  factor  is 

important  : : : : 

: : : : unimportant 

2 


Processing  of  requests  for  changes  to  existing  systems 


fast  :::::: 

slow 

timely  ::::::: 

untimely 

simple  ::::::: 

complex 

flexible  ::::::: 

j rigid 

satisfactory  ; : : : : : : 

j unsatisfactory 

To  me,  this  factor  is 

Important  ::::::: 

unimportant 

3 

*EDP:  Electronic  Data  Processing 


4 


: unimportant 


rea 


definite  ::::::::  uncertain 


good  : : : : : : : : bad 

satisfactory  : : : : : : : : unsatisfactory 

To  me,  this  factor  is 

important  : : : : : : : : unimportant 


6 


Timeliness  of  output  information 


To  me 


timely  : 
reasonable  : 


consistent  : 


punctual 
satisfactory 
, this  factor  is 
important 


: untimely 
: unreasonable 
: inconsistent 
: tardy 

: unsatisfactory 

: unimportant 


t * 


/ 
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Charge-back  method  of  payment  for  services 


just 
reasonable 
consistent 
known  : 
satisfactory  : 


unjust 

unreasonable 
inconsistent 
: unknown 
: unsatisfactory 


To  me,  this  factor  is 


important  : : : : : : : : unimportant 


Perceived  utility  (worth  versus  cost) 


high 
positive 
sufficient  : 
useful  : 
satisfactory  : 


low 

negative 

Insufficient 

useless 

; unsatisfactory 


To  me,  this  factor  is 


Important  : : : 


6 


unimportant 


Vendor  support  of  hardware  and  software 

21  • skilled  : : 

• • 

• • 

• • 

• • 

• 

• 

bungling 

sufficient  : : 

• • 

• » 

• • 

• • 

• 

• 

unsufficient 

eager  : : 

• • 

• • 

• • 

• • 

• 

• 

indifferent 

consistent  : : 

• • 

• • 

• • 

• • 

• 

• 

inconsistent 

satisfactory  : : 

• • 

• ♦ 

• • 

• • 

• 

• 

unsatisfactory 

To  me,  this  factor  is 

important  : : 

• • 

• • 

• • 

• 

• 

unimportant 

Computer  language  used  to 

interact  with  systems 

easy  : : 

• • • • • 

• • • • • 

difficult 

easy-to-use  : : 

• • • • • 

• • • • • 

hard-to-use 

satisfactory  : : 

• • • • • 

unsatisfactory 

To  me,  this  factor  is 

important  : : 

• • • • • 

• • • • • 

unimportant 

7 

pleased  : 

high  : 
definite  : 


optimistic 
satisfactory 
To  me,  this  factor  is 
important 


: displeased 
: low 

: uncertain 
: pessimistic 
: unsatisfactory 


: unimportant 


12 


Correction  of  errors 


fast 
superior 
complete 
simple  : 


slow 
inferior 
: incomplete 


satisfactory 


13 


Security  of  data 


secure 
good 
definite 
complete 
satisfactory 
To  me,  this  factor  is 
important  : 


unsecure 

bad 

uncertain 

incomplete 

unsatisfactory 

unimportant 


14 


Degree  of  EDP  training  provided  to  users 


complete  : 
sufficient  : 


high 
superior 
satisfactory 
To  me,  this  factor  is 


incomplete 

insufficient 

low 

inferior 

unsatisfactory 


important  : 


: unimportant 
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A 


15  Understanding  of  systems 


high 


sufficient  : 
complete 


easy  : 


satisfactory  : : 

To  me,  this  factor  is 


insufficient 

incomplete 


unsatisfactory 


important  : : : : : : : : unimportant 


16  Feeling  of  participation 


positive  : 
encouraged  : 
sufficient 
involved 
satisfactory  : 


negative 

repelled 

insufficient 

uninvolved 

unsatisfactory 


To  me,  this  factor  is 


important  : : : : : : : : unimportant 
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17  Currency  (up-to-dateness)  of  the  output  information 


timely 


adequate 


reasonable 


satisfactory 


untimely 


inadequate 


unreasonable 


unsatisfactory 


To  me,  this  factor  is 


important  : : : : : : : : unimportant 


18  Attitude  of  the  EDP  staff 


user-oriented  : 


cooperative  : 


courteous 


positive 


satisfactory  : 


: self-centered 


belligerent 


: discourteous 


: negative 


: unsatisfactory 


To  me,  this  factor  is 


important  : : : : : j : : unimportant 


11 


19 


Reliability  of  output  information 


consistent 

! 

high 
superior 
sufficient  : 


satisfactory  : 


To  me,  this  factor  is 


important 


20 


Top  management  involvement  in  EDP  activities 


strong  : 

consistent  : 

good  : 

significant  : : 

satisfactory  : : : : 
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inconsistent 

low 

inferior 

insufficient 

unsatisfactory 


unimportant 


weak 

inconsistent 

bad 

insignificant 
: unsatisfactory 


To  me,  this  factor  is 


important  : : : : : : : unimportant 


12 


1 


! 


» 
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21 


Format  of  output 


23  Determination  of  priorities  for  allocation  of  EDP  resources 


fair  : 

• 

• 

• 

• 

• • 

• • 

: unfair 

consistent  : 

• 

• 

• 

• 

• • 

• • 

: inconsistent 

just  : 

• 

• 

• 

• 

• • 

• • 

: unjust 

precise  : 

• 

• 

• 

• 

• • 

* • 

: vague 

satisfactory  : 

To  me,  this  factor  is 

• 

• 

• 

• 

• • 

• • 

: unsatisfactory 

important  : . 

• 

• 

• 

• 

• • 

• • 

: unimportant 

24  Convenience  of  access  (to  utilize  the  computer  capability) 


convenient  : : : : : : : : inconvenient 

good  : : : : : : : : bad 

easy  : : : : : : : : difficult 

efficient  : : : : : : : : inefficient 

satisfactory  : : : : : * : : : unsatisfactory 

To  me,  this  factor  is 

important  : : : : : : : : unimportant 


14 
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25  Relevancy  of  output  information  (to  intended  function) 


useful  : : : : : : : : useless 

relevant  : : : : : : : : irrelevant 

clear  : : v : : : : : : hazy 

good  : : : : : : : : bad 

satisfactory  : : : : : : : : unsatisfactory 

To  me,  this  factor  is 

important  : : : : : : : : unimportant 


26  Volume  of  output  information 


concise  : : 

: : : : redundant 

sufficient  : : : 

: : : : : insufficient 

necessary  : : : 

: : : : : unnecessary 

reasonable  : : : 

: : : : : unreasonable 

satisfactory  : : : 

: : : : : unsatisfactory 

To  me,  this  factor  is 

important  : : 

: : : : unimportant 

27  Personal  job  effects  resulting  from  the  computer-based  support 


liberating 

significant 


valuable 

satisfactory 


inhibiting 

Insignificant 


worthless 

unsatisfactory 


To  me,  this  factor  is] 


important  : : : : : : : : unimportant 


28  Accuracy  of  output  information 


accurate  : 


high 
consistent 
sufficient  : 


satisfactory  : : : 


To  me,  this  factor  is 


: inaccurate 


inconsistent 
insufficient 
: unsatisfactory 


Important  : : : : : : : : unimportant 


17 


: unimportant 


/ 
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31 


Organizational  position  of  the  EDP  function 


, appropriate  : 

• • • • 

• • • • 

• • • 

• • • 

strong  : 

• • • • 

• • • • 

• • • 

• • • 

clear  : 

• • • • 

• • • • 

• • • 

• • • 

progressive  : 

• • • • 

• • • • 

• • • 

• • • 

satisfactory  : 

• • • • 

• • • • 

• • • 

• • • 

To  me,  this  factor  is 

important  : : : : : 


inappropriate 

weak 

hazy 

regressive 

unsatisfactory 

unimportant 


32 


Time  required  for  new  systems  development 


short 

dependable  : 
reasonable 
acceptable 
satisfactory  : 


To  me,  this  factor  is 


important 


long 

urdependable 
unreasonable 
: unacceptable 
unsatisfactory 

unimportant 


18 


» 
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Personal  control  of  EDP  service  received 


high  : 

sufficient  : 

precise  : 

strong  : : 

satisfactory  : 


To  me,  this  factor  is 


important  : : : : : : : : unimportant 


34 


Schedule  of  recurring  output  products  and  services 


good  : 

regular  : 

reasonable  : 

acceptable  : 

satisfactory  : 

pTo  me,  this  factor  is| 


bad 

irregular 
unreasonable 
unacceptable 
•.  unsatisfactory 


Important  ::::::::  unimportant 


19 
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35  Documentation 


' clear  : : : : : : : : hazy 

available  : : : : : : : : unavailable 

complete  : : : : : : : : incomplete 

current  : : : : : : : : obsolete 

satisfactory  : : : : : : : : unsatisfactory 

To  me,  this  factor  is 

important  : : : : : : : : unimportant 


36  Completeness  of  the  output  information 


complete  :::::: 

: incomplete 

consistent  :::::: 

: inconsistent 

sufficient  : : : : : : 

: : insufficient 

adequate  : : : : : - : 

: : inadequate 

satisfactory  :::::: 

: unsatisfactory 

To  me,  this  factor  Is 

important  : : : : : 

: : unimportant 

20 


37  Technical  competence  of  the  EDP  staff 


current 


sufficient  : 
superior  : 


high 

satisfactory  : 


To  me,  this  factor  is 


obsolete 

Insufficient 


inferior 


unsatisfactory 


important  : : : : : : : : unimportant 


38  Flexibility  of  systems 


flexible  : 
versatile  : 


sufficient  : 


high  : : 

satisfactory  : 

To  me,  this  factor  is 


rigid 
limited 
insufficient 
: low 

: unsatisfactory 


important  : : : : : : : : unimportant 


21 


Integration  (automated  sharing  of  information)  of  system  data  bases 


complete  : 
sufficient 
successful 
good 

satisfactory  : 


incomplete 

insufficient 

unsuccessful 

bad 

unsatisfactory 


. . 5 


t 


► » 


273 


Appendix  4.2 

INSTRUCTIONS  FOR  ADJECTIVE  PAIR  SELECTION 

The  selection  of  the  adjective  pairs  for  factor  scales 
was  based  on  a two-phase  Delphi  type  procedure.  The 
following  instructions  and  forms  for  both  phases  were 
completed  by  five  university  professors. 
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PURPOSE:  The  purpose  of  this  exercise  is  the  selection  of 
bipolar  adjective  pairs  which  may  be  used  to  evaluate  each 
concept  that  is  listed.  Each  concept  is  related  to  a part- 
icular aspect  of  the  computer-based  information  products  and 
services  provided  to  a user  within  an  organization. 

PROCEDURE:  The  adjective  pairs  that  are  selected  for  each 
concept  should  be  highly  germane  to  that  individual  aspect 
with  a clear  polarity  of  "gocdness"  or  "badness".  There  is 
absolutely  no  limit  to  the  number  of  times  that  an  adjective 
pair  may  be  selected  for  different  concepts.  The  list  of  ad- 
jective pairs  that  have  been  used  in  other  studies  are  provided 
only  as  a guide.  Any  adjective  pair  that  you  deem  germane  for 
evaluating  a concept  may  be  selected.  Most  adjectives  can  be 
negatively  paired  by  the  use  of  prefixes  such  as  "un-" , "in-", 
"dis-",  etc.  The  steps  to  be  followed  are: 

1.  Read  the  list  of  adjective  pairs  to  familiarize 
yourself  with  words  that. have  been  selected  in 
other  studies. 

2.  For  each  concept  listed,  select  or  formulate  FOUR 
bipolar  adjective  pairs  which  are  indicative  of 
evaluation  for  that  concept. 

3.  Write  the  selected  adjective  pairs  in  the  spaces 
provided  for  each  concept.  If  the  pair  is  a duplicate 
of  a pair  listed  in  the  guide,  the  associated  number 
may  be  written  into  the  space.  An  example  is  shown 
below: 

CONCEPT 


#*This  pair  is  not  listed  in  the  guide 
This  pair  is  an  altered  form  of  4 and  34 

4.  This  exercise  is  completed  when  selections  have  been 
made  for  each  of  the  concepts  listed. 


SEQUEL » The  adjective  pairs  selected  for  each  concept  by 
every  individual  respondent  will  be  assembled.  These  group- 
ings will  be  redistributed  for  a second,  more  confined,  judg- 
ment. . At  that  time,  you  will  be  asked  to  select  the  top  four 
adjective  pairs  that  may  be  used  to  evaluate  the  concepts. 
This  sequel  step  should  be  much  less  time  consuming  than  the 
initial  selection  process.  Your  cooperation  is  gratefully 
appreciated . 


ADJECTIVE  PAIRS  USED  IN  OTHER 
SEMANTIC  DIFFERENTIAL  STUDIES 


277 
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1. 

2. 

3. 

5. 

6. 

7. 

8. 

9. 

10. 
11. 
12. 

13. 

14. 

15. 

16. 

17. 

18. 

19. 

20. 
21. 
22. 

23. 

24. 

25. 


good  - bad 

optimistic  - pessimistic 
complete  - incomplete 
timely  - untimely 
sociable  - unsociable 
kind  - cruel 
harmonious  - dissonant 
selfish  - unselfish 
willing  - unwilling 
superior  - inferior 
sufficient  - insufficient 
voluntary  - compulsory 
assenting  - dissenting 
skilled  - bungling 
successful  - unsuccessful 
high  - low 

meaningful  - meaningless 
lucid  - obscure 
important  - trivial 
useful-  useless 
progressive  - regressive 
true  - false 
positive  - negative 
candid  - deceitful 
wise  - foolish 


, _ r|  | | ^j||fA|  ■ , |m 


26.  strong  - weak 

27.  concise  - diffuse 

28.  fast  - slow 

29.  complex  - simple 

30.  stable  - unstable 

31.  cautious  - rash 

32.  loyal  - disloyal 

33.  courteous  - discourteous 

34.  early  - late 

35.  interesting  - boring 

36.  eager  - indifferent 

37.  destructive  - productive 

38.  consistent  - inconsistent 

39.  regular  - irregular 

40.  chaotic  - ordered 

41.  clear  - hazy 

42.  concentrated  - diffuse 

43.  definite  - uncertain 

44.  emotional  - rational 

45.  expensive  - cheap 

46.  fair  - unfair 

47.  pleasant  - unpleasant 

48.  powerful  - weak 

49.  vague  - precise 

50.  valuable  - worthless 


Charge-back  method  of  payment  for  services 


Perceived  utility  (worth  versus  cost) 


Vendor  support  of  hardware  and  software 


Language  used  to  interact  with  systems 


Expectations  concerning  computer-based  support 


Error  recovery 


Security  of  data 
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Degree  of  training 


Understanding  of  systems 


Feeling  of  participation 


Currency  of  output  information 


Attitude  of  the  EDP  staff 


Reliability  of  output  information 


Top  management  involvement 


' 


km 


Format  of  output 


Response/tumaround  time 


Priorities  determination  for  EDP  resource  allocation 


Convenience  of  access  (to  utilize  the  computer  capability) 


Relevancy  of  output  information  (to  intended  function) 


Volume  of  output  information 


Job  effects  attributed  to  computer-based  support 


Documentation 


Completeness  of  the  output  information 


Technical  competence  of  the  ED?  staff 


Flev"  ility  of  systems 


Integration  of  systems 
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PURPOSE:  The  purpose  of  this  exercise  is  to  select  a final  set 

of  bipolar  adjective  pairs  that  are  germane  to  evaluating  differ- 
ent aspects  of  the  computer-based  information  products  and  ser- 
vices that  are  provided  to  a user.  It  is  crucial  that  these 
adjective  pairs  be  indicative  of  a "good"  or  "baa"  state  of 
each  concept  from  the  perspective  of  the  user.  In  other  words, 
if  the  "positive  pole"  of  the  adjective  pair  describes  the 
concept,  the  user  is  more  satisfied  than  if  the  "negative  pole" 
describes  the  concept.  The  adjective  pairs  snould  not  be  selected 
to  indicate  how  important  the  concept  is  to  the  user.  Each  ad- 
jective pair  should  be  appropriate  for  evaluating  the  concept 
in  a word  completion  format,  such  as,  "The  CONCERT  is  ( adjective) " 


PROCEDURES:  Each  concept  that  is  listed  is  followed  by  a set 
of  bipolar  adjective  pairs  that  were  formulated  during  the 
initial  phase.  The  selection  process  is  confined  to  this  set 
of  adjective  pairs.  The  steps  to  be  followed  are: 

1.  For  each  concept  listed,  select  the  FOUR  bipolar 
adjective  PAIRS  that  are  most  indicative  of  evaluation 
for  that  concept. 

2.  Flace  an  "X"  to  the  immediate  left  of  the  selected 
adjective  pairs. 

3.  For  each  adjective  pair  selected,  place  a "+"  over  the 
adjective  that  indicates  the  positive  state  of  that 
concept  for  the  user.  The  "+"  indicates  the  positive 
direction  of  the  adjective  pair  for  that  concept.  An 
example  is  shown  below: 

CONCEPT 

X good  - bad  complete  - incomplete 

fair  - unfair  >Cfast  - slow 

X clear  - hazy  X superior  - inferior 

4.  The  exercise' is  completed  when  selections  have  been 
made  for  each  of  the  concepts  listed.  Thank  you  very 
much  for  your  cooperation. 


nL*-^ 


CONCEPTS 


Relationship  with  the  EDP  staff 


harmonious  - dissonant 
productive  - destructive 
cooperative  - competitive 
pleasant  - unpleasant 
valuable  - worthless 
cooperative  - uncooperative 
professional  - unprofessional 
frequent  - infrequent 


superior  - inferior 
good  - bad 

friendly  - unfriendly 
cooperative  - belligerent 
helpful  - bothersome 
formal  - informal  . 
candid  - deceitful 


Processing  of  requests  for  changes  to  existing  systems 


fast  - slow 
complex  - simple 
chaotic  - ordered 
willing  - unwilling 
reasonable  - unreasonable 
good  - bad 
timely  - untimely 
eager  - indifferent 


rigid  - flexible 
difficult  - easy 
skilled  - bungling 
successful  - unsuccessful 
reliable  - unreliable 
timely  - late 
early  - late 


Mode  of  interface  for  input/output  with  the  EDP  center 
convenient  - inconvenient  good  - bad 

appropriate  - inappropriate  favorable  - unfavorable 

proper  - improper  complete  - incomplete 

useful  - useless  clear  - hazy 


courteous  - discourteous 
efficient  - inefficient 
meaningful  - meaningless 
chaotic  - ordered 
frequent  - infrequent 
regular  - irregular 


pleasant  - unpleasant 
organized  - disorganized 
defined  - undefined 
complex  - simple 
enjoyable  - unen joyable 


J 
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Organizational  competition  with  the  EDP  unit 
productive  - destructive  positive  - negative 

friendly  - unfriendly  fair  - unfair 

high  - low  viscious  - friendly 

useful  - destructive  harmonious  - dissonant 

strong  - weak  good  - bad 

significant  - insignificant  clear  - hazy 

emotional  - rational 


Confidence  in 

high  - low 
positive  - negative 
optimistic  - pessimistic 
emotional  - rational 
superior  - inferior 
reserved  - unreserved 

Timeliness  of  output 
timely  - untimely 
consistent  - inconsistent 
meaningful  - meaningless 
stable  - unstable 
reasonable  - unreasonable 
early  - late 

Charge-back  method  of 
just  - unjust 
fair  - unfair 
good  - bad 

progressive  - regressive 
consistent  - inconsistent 
reasonable  - unreasonable 
voluntary  - compulsory 


ystems 

stable  - unstable 
strong  - weak 
definite  - uncertain 
complete  - incomplete 
good  - bad 

information 

appropriate  - unappropriate 
good  - bad 
useful  - useless 
fast  - slow 
regular  - irregular 
punctual  - tardy 

payment  for  services 
clear  - hazy 
regular  - irregular 
proper  - improper 
destructive  - productive 
known  - unknown 
meaningful  - meaningless 
expensive  - cheap 
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Perceived  utility  (worth  versus  cost) 


expensive  - cheap 
high  - low 

sufficient  - insufficient 
definite  - uncertain 
good  - bad 
clear  - hazy 

significant  - insignificant 


positive  - negative 
valuable  - worthless 
regular  - irregular 
superior  - inferior 
known  - unknown 
useful  - useless 


Vendor  support  o 

good  - bad 
skilled  - bungling 
strong  - weak 
eager  - indifferent 
reasonable  - unreasonable 
sufficient  - insufficient 


hardware  and  software 

timely  - untimely 
superior  - inferior 
complete  - incomplete 
consistent  - inconsistent 
known  - unknown 


Language  used  to  interact  with  systems 


simple  - complex 
complete  - incomplete 
sufficient  - insufficient 
valuable  - worthless 
superior  - inferior 
known  - unknown 
difficult  - easy 
vague  - precise 

Expect,  ions  concerning 
contented  - frustrated 
pleased  - displeased 
successful  - unsuccessful 
stable  - unstable 
definite  - uncertain 
vague  - precise 


appropriate  - inappropriate 
good  - bad 
clear  - hazy 
powerful  - weak 
reasonable  - unreasonable 
easy-to-use  - hard-to-use 
compatible  - incompatible 
universal  - variable 

computer-based  support 
high  - low 

optimistic  - pessimistic 
useful  - useless 
chaotic  - ordered 
clear  - hazy 
emotional  - rational 


Error  recovery 


fast  - slow 
definite  - uncertain 


complete  - incomplete 
sufficient  - insufficient 
superior  - inferior 
frequent  - infrequent 
clear  - hazy 


skilled  - bungling 
good  - bad 
timely  - untimely 
known  - unknown 
complex  - simple 
expensive  - cheap 


Security  of  data 


secure  - unsecure 
definite  - uncertain 
successful  - unsuccessful 
useful  - useless 
good  - bad 

consistent  - inconsistent 
vague  - precise 


complete  - incomplete 
strong  - weak 
high  - low 
stable  - unstable 
known  - unknown 
cautious  - rash 


Degree  of  training 

complete  - incomplete  regular  - irregular 

high  - low  superior  - inferior 

valuable  - worthless  complex  - simple 

sufficient  - insufficient  strong  - weak 

skilled  - bungling 

Understanding  of  systems 
high  - low  clear  - hazy 

complete  - incomplete  good  - bad 

sufficient  - insufficient  voluntary  - compulsory 

chaotic  - ordered  hard  - easy 

superior  - inferior 
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Feeling 

active  - passive 
positive  - negative 
superior  - inferior 
strong  - weak 
sufficient  - insufficient 
reasonable  - unreasonable 
good  - bad 
vague  - precise 

Currency  of 

old  - new 
good  - bad 
high  - low 

adequate  - inadequate 
superior  - inferior 

Attitude  of 
user-oriented  - self-centered 
willing  - unwilling 
sociable  - unsociable 
valuable  - worthless 
courteous  - discourteous 
friendly  - unfriendly 
cooperative  - belligerent 
positive  - negative 
emotional  - rational 


f participation 

complete  - incomplete 
encouraged  - repelled 
high  - low 
definite  - uncertain 
informed  - uninformed 
involved  - uninvolved 
voluntary  - compulsory 
valuable  - worthless 

output  information 

timely  - untimely 
sufficient  - insufficient 
stable  - unstable 
reasonable  - unreasonable 
early  - late 

the  SDP  staff 

loyal  - disloyal 
eager  - indifferent 
harmonious  - dissonant 
destructive  - productive 
good  - bad 

helpful  - bothersome 
pleasant  - unpleasant 
assenting  - dissenting 


Reliability  of 
consistent  - inconsistent 
good  - bad 

valuable  - worthless 
superior  - inferior 
reasonable  - unreasonable 


output  information 

complete  - incomplete 
high  - low 
stable  - unstable 
regular  - irregular 
sufficient  - insufficient 


» 


A 
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Top  management 

strong  - weak 
positive  - negative 
complete  - incomplete 
constructive  - destructive 
helpful  - bothersome 
good  - bad 

meaningful  - meaningless 
destructive  - productive 


involvement 

progressive  - regressive 
consistent  - inconsistent 
successful  - unsuccessful 
proper  - improper 
certain  - uncertain 
voluntary  - compulsory 
significant  - insignificant 


Format  of  output 

clear  - hazy  ordered  - chaotic 

good  - bad  simple  - complex 

sufficient  - insufficient  useful  - useless 

superior  - inferior  liked  - hated 

readable  - unreadable  pleasant  - unpleasant 


Response/turnaround  time 

fast  - slow  consistent  - inconsistent 

timely  - untimely  good  - bad 

regular  - irregular  early  - late 

enough  - insufficient  reasonable  - unreasonable 

definite  - uncertain 


Priorities  determination  for  EDP  resource  allocation 


fair  - unfair 
just  - unjust 
proper  - improper 
successful  - unsuccessful 
consistent  - inconsistent 
good  - bad 
clear  - hazy 
political  - apolitical 


favorable  - unfavorable 
progressive  - regressive 
sufficient  - insufficient 
useful  - useless 
chaotic  - ordered 
known  - unknown 
vague  - precise 


Convenience  of  access  (to  utilize  the  computer  capability) 


convenient  - inconvenient 
complete  - incomplete 
sufficient  - insufficient 
consistent  - inconsistent 
superior  - inferior 
known  - unknown 
fast  - slow 
fair  - unfair 


difficult  - easy 
favorable  - unfavorable 
high  - low 

pleasant  - unpleasant 
good  - bad 

efficient  - inefficient 
complex  - simple 


Relevancy  of  output  information  (to  intended  function) 


complete  r incomplete 
superior  - inferior 
high  - low 
useful  - useless 
clear  - hazy 

meaningful  - meaningless 
timely  - untimely 


relevant  - irrelevant 
valuable  - worthless 
good  - bad 

sufficient  - insufficient 
known  - unknown 
vague  - precise 
destructive  - productive 


Volume  of  output 
redundant  - concise 
necessary  - unnecessary 
high  - low 

destructive  - productive 
sufficient  - insufficient 
liked  - hated 
complete  - incomplete 
concentrated  - diffuse 


information 
good  - bad 
sparse  - sufficient 
useful  - useless 
valuable  - worthless 
reasonable  - unreasonable 
known  - unknown 
consistent  - inconsistent 


Job  effects  attributed 
positive  - negative 
inhibiting  - liberating 
good  - bad 

successful  - unsuccessful 
significant  - insignificant 
clear  - hazy 


to  computer-based  support 
valuable  - worthless 
destructive  - productive 
sufficient  - insufficient 
useful  - useless 
interesting  - boring 


Accuracy  of  output  information 
accurate  - inaccurate  good  - bad 

consistent  - inconsistent  definite  - uncertain 

high  - low  stable  - unstable 

known  - unknown  reasonable  - unreasonable 


known  - unknown 
sufficient  - insufficient 


clear  - hazy 


Precision  of  output  formation 


vague  - precise 
sufficient  - insufficient 
complete  - incomplete 
useful  - useless 
consistent  - inconsistent 
superior  - inferior 


definite  - uncertain 
known  - unknov.n 


Communication  with  the  ED P staff 
complete  - incomplete  successful  - unsuccessful 

simple  - complex  good  - bad 

harmonious  - dissonant  destructive  - productive 

cl~ar  - hazy  pleasant  - unpleasant 

courteous  - discourteous  high  - low- 

interesting  - dull  willing  - unwilling 

vague  - precise  frequent  - infrequent 

lucid  - obscure  meaningful  - meaningless 

Organizational  position  of  the  EDP  function 


appropriate  - inappropriate 

just  - unjust 

good  - bad 

high  - low 

clear  - hazy 

fair  - unfair 

strong  - weak 


positive  - negative 
progressive  - regressive 
superior  - inferior 
vague  - precise 
known  - unknown 
powerful  - weak 
definite  - uncertain 
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Time  required  for  new  systems  development 


long  - short 

reasonable  - unreasonable 

high  - low 

definite  - uncertain 

fast  - slow 

known  - unknown 

voluntary  - compulsory 


dependable  - undependable 

good  - bad 

clear  - hazy 

vague  - precise 

early  - late 

acceptable  - unacceptable 


S 

t 
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Feeling  of  control 


high  - low 

complete  - incomplete 
definite  - uncertain 
vague  - precise 
defined  - undefined 


sufficient  - insufficient 
strong  - weak 
powerful  - weak 
chaotic  - ordered 
fair  - unfair 


Schedule  of  recurring  output  products  and  services 


timely  - untimely 
fair  - unfair 
good  - bad 
useful  - useless 
early  - late 

organized  - disorganized 
reasonable  - unreasonable 
sufficient  - insufficient 


chaotic  - ordered 
regular  - irregular 
skilled  - bungling 
destructive  - productive 
fast  - slow 
known  - unknown 
acceptable  - unacceptable 


clear  - hazy 
complete  - incomplete 
proper  - improper 
timely  - untimely 
easy  - hard 

available  - unavailable 


Documentation 

useful  - useless 
current  - obsolete 
valuable  - worthless 
good  - bad 

meaningful  - meaningless 
accurate  - inaccurate 
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Completeness  of  the  output  information 


complete  - incomplete 
sufficient  - insufficient 
consistent  - inconsistent 
high  - low 


comprehensive  - sparse 
good  - oad 

adequate  - inadequate 
superior  - inferior 


Technical  competence  of  the  EDP  staff 


current  - obsolete 
skilled  - bungling 
consistent  - inconsistent 
strong  - weak 
known  - unknown 
sufficient  - insufficient 

Flexibility 

flexible  - rigid 
good  - bad 

flexible  - inflexible 
clear  - hazy 
definite  - uncertain 
known  - unknown 
sufficient  - insufficient 
powerful  - weak 


superior  - inferior 
complete  - incomplete 
definite  - uncertain 
good  - bad 
high  - low 
powerful  - weak 

systems 

versatile  - limited 
adept  - clumsy 
progressive  - regressive 
high  - low 
superior  - inferior 
reasonable  - unreasonable 
fair  - unfair 


Integration  o 
complete  - incomplete 
valuable  - worthless 
high  - low 

consistent  - inconsistent 
powerful  - weak 
known  - unknown 
superior  - inferior 
political  - apolitical 


systems 

productive  - destructive 

successful  - unsuccessful 

sufficient  - insufficient 

strong  - weak 

good  - bad 

useful  - useless 

vague  - precise 
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Appendix  4.3 

ADJECTIVE  PAIR  SELECTIONS 

The  following  adjective  pairs  were  selected  to  form  the 
evaluation  scales  for  the  associated  factors.  The  number  of 
judges  selecting  the  individual  adjective  pairs  is  shown  in 
parentheses . 


I 


ADJECTIVE  PAIR  SELECTIONS 


Relationship  with  the  EDP  staff 
harmonious  - dissonant (5) 
cooperative-uncooperative (3) 


good  - bad(2) 
candid  - deceitful (2) 


Processing  of  requests  for  changes  to  existing  systems 
fast  - slow(4)  rigid  - flexible(2) 

complex  - simple (3)  timely  - untimely (2) 

Mode  of  interface  for  input/output  with  the  EDP  center 
convenient  - inconvenient (4)  clear  - hazy (3) 

efficient  - inefficient (2)  organized  - disorganized(4) 

Organizational  competition  with  the  EDP  unit 

productive  - destructive (3)  harmonious  - dissonant (4) 

high  - low(3)  emotional  - rational (3) 


Confidence  in  systems 
high  - low(5) 
reserved  - unreserved(2) 

Timeliness  of  output  information 
timely  - untimely(3) 
consistent  - inconsistent (4) 


strong  - weak (3) 
good  - bad(3) 


punctual  - tardy (3) 
reasonable  - unreasonable (2) 


Charge-back  method  of  payment  for  services 

just  - unjust (3)  known  - unknown(3) 

consistent  - inconsistent(3)  reasonable  - unreasonable (2) 

Perceived  utility  (worth  versus  cost) 


high  - low(3) 

sufficient  - insuf ficient(2) 


positive  - negative(3) 
useful  - useless (2) 


Vendor  support  of  hardware  and  software 

skilled  - bungling (3)  consistent 

eager  - indifferent (3)  sufficient 


inconsistent(3; 
insufficient (3) 


Language  used  to  interact  with  systems 

simple  - complex(4)  powerful  - weak(2) 

difficult  - easy(3)  easy-to-use  - har d-to-use(4) 

Expectations  concerning  computer-based  support 
pleased  - displeased(2)  high  - low(3) 

definite  - uncertain(3)  optimistic  - pessimistic(3) 


298 


» 


t 


♦ 


Error  recovery 
fast  - slow(3) 
complete  - incomplete (2) 

Security  of  data 

secure  - unsecure(3) 
definite  - uncertain(2) 

Degree  of  training 

complete  - incomplete (3) 
high  - low(3) 

Understanding  of  systems 
high  - low(3) 
complete  - incomplete (4) 

Feeling  of  participation 
positive  - negative(3) 
sufficient  - insufficient (3) 

Currency  of  output  information 
good  - bad(2) 
adequate  - inadequate(2) 

Attitude  of  the  EDP  staff 

user-oriented  - self-centered(5) 
cooperative  - belligerent (4) 

Reliability  of  output  information 
consistent  - inconsistent (4) 
superior  - inferior (2) 

Top  management  involvement 
strong  - weak (3) 
significant  - insignificant (2) 

Format  of  output 
good  - bad(2) 
readable  - unreadable (5) 

Response/ turnaround  time 
fast  - slow(4) 

reasonable  - unreasonable(3) 


complex  - simple (3) 
superior  - inferior (2) 


complete  - incomplete(2) 
good  - bad(3) 


superior  - inferior(3) 
sufficient  - insuff icient(5) 


hard  - easy (4) 

sufficient  - insufficient (5) 


encouraged  - repelled (4) 
involved  - uninvolved (2) 


timely  - untimely(5) 
reasonable  - unreasonable(3) 


courteous  - discourteous (2) 
positive  - negative(2) 


high  - low(3) 

sufficient  - insufficient(5) 


consistent  - inconsistent(5) 
good  - bad(2) 


simple  - complex (3) 
useful  - useless(4) 


consistent  - inconsistent(3) 
good  - bad(3) 


Priorities  determination  for  EDP  resource  allocation 
fair  - unfair(5)  just  - unjust(5) 

consistent  - inconsistent (4)  vague  - precise(2) 


Convenience  of  access  ( to  utilize  the  computer  capability  ) 
convenient  - inconvenient (3)  difficult  - easy (4) 

good  - bad(2)  efficient  - ineff icient(3) 

Relevancy  of  output  information  (to  intended  function) 
useful  - useless (2)  good  - bad(2) 

relevant  - irrelevant (3)  clear  - hazy (3) 


Volume  of  output  information 
redundant  - concise (2) 
necessary  - unnecessary (2) 


reasonable  - unreasonable (3) 
sufficient  - insufficient (5) 


Job  effects  attributed  to  computer-based  support 

inhibiting  - liberating(2)  valuable  - worthless (5) 

significant  - insignificant (3)  good  - bad(4) 


Accuracy  of  output  information 
accurate  - inaccurate (3) 
sufficient  - insufficient (4) 

Precision  of  output  information 
sufficient  - insufficient (4) 
consistent  - inconsistent (4) 

Communication  with  the  EDP  staff 
harmonious  - dissonant (4) 
destructive  - productive (3) 


consistent  - inconsistent(4) 
high  - low(4) 


high  - low(2) 
definite  -•  uneertain(2) 


vague  - precise(2) 
meaningful  - meaningless (3) 


Organizational  position  of  the  EDP  function 

appropriate  - inappropriate (5)  clear  - hazy (3) 
progressive  - regressive(2)  strong  - weak(2) 

Time  required  for  new  systems  development 

long  - short (2)  dependable  - undependable (4) 

reasonable  - unreasonable (4)  acceptable  - unacceptable(4) 


Feeling  of  control 
high  - low(3) 

sufficient  - insufficient (4) 


strong  - weak (2) 
vague  - precise(3) 


Schedule  of  recurring  output  products  and  services 
regular  - irregular (3)  good  - bad(2) 

reasonable  - unreasonable (4)  acceptable  - unacceptable(3) 


Documentation 
clear  - hazy(4) 
complete  - incomplete (4) 


current  - obsolete(2) 
available  - unavailable(3) 


r 
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Completeness  of  the  output  information 

complete  - incomplete(3)  adequate  - inadequate(4) 

sufficient  - insuf ficient(5)  " J * — 


consistent  - inconsistent(2) 


Technical  competence  of  the  EDP  staff 

current  - obsolete(3)  superior  - inferior (2) 

sufficient  - insuf ficient (4)  high  - low(3) 


Flexibility  of  systems 
flexible  - rigid(3) 
sufficient  - insuf ficient (4) 


high  - low (2) 
versatile  - limited(2) 


Integration  of  systems 
complete  - incomplete (3) 
sufficient  - insufficient (3) 


successful  - unsuccessful(3) 
good  - bad(2) 
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Appendix  5.1 

FORTRAN  PROGRAM  FOR  DATA  SUMMARIZATION 

The  following  computer  listing  illustrates  the  program 
used  to  summarize  the  data  from  the  completed  measurement 
instruments.  A constant  of  +4.0  was  added  to  each  of  the 
sever,  response  interval  values  to  change  the  range  of  -3  to 
+3  to  the  range  +1  to  +7,  thereby  eliminating  negative 
numbers  during  the  input  process.  The  program  was  written 
in  the  FORTRAN  IV  language  and  executed  on  a UNIVAC  1110. 

No  attempt  was  made  to  optimize  the  program  execution. 
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Appendix  5.2 


FORTRAN  PROGRAM  FOR  ANALYSIS  OF  VARIANCE 


The  following  computer  listing  illustrates  the  program 
used  to  compute  the  analysis  of  variance  and  to  calculate  a 
reliability  coefficient  for  each  factor.  The  program  also 
calculated  the  importance  scale  means  and  the  scale  means 
for  the  satisfied  and  dissatisfied  user  groups.  The  differ- 
ence in  means  was  calculated  as  the  range.  The  item  (scale) 
scores  shown  in  the  sample  output  include  a constant  of  +4.0 
which  was  added  to  eliminate  negative  numbers  during  the 
input  process.  The  program  was  written  in  the  FORTRAN  IV 
language  and  executed  on  a UNIVAC  1110.  No  attempt  was  made 
to  optimize  the  program  execution. 
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Appendix  5 . 3 

SAMPLE  FACTOR  ANALYSIS  OUTPUT 

The  following  computer  listing  illustrates  a sample 
output  from  the  SPSS  package  programs  for  factor  analysis. 
The  program  was  executed  on  a UNIVAC  1110.  The  variables 
labeled  V001,  V002,  V003,  and  V004  correspond  to  the  four 
factor  one  scale  values,  respectively.  These  values  include 
a constant  of  +4.0  which  was  added  to  eliminate  negative 
numbers  during  the  input  process. 


Kl  i-l  ABILITY 


SAMPLE  OUTPUT 
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PURPOSE:  The  purpose  of  this  exercise  is  to  identify  middle  management 
users  of  computer-based  information  products  and  serve ies  who  may  be  classi- 

I fied  as  either  (1)  extremely  satisfied,  (2)  extremely  dissatisfied,  or  (3) 

slightly  satisfied  or  dissatisfied  with  the  computer-based  support  that  they 
now  receive  in  their  present  job  position.  These  three  groups  will  be  used 
to  test  the  validity  of  a measurement  instrument  designed  to  measure  user 
satisfaction. 

t PROCEDURE:  The  following  procedures  should  be  followed  in  making  the  user 

identifications.  The  classifications  should  reflect  how  the  users  feel  about 
the  support  that  they  receive;  not  how  you  feel  about  their  support. 

1.  List  the  middle  management  users  whom: 

(a ) you  know  wel 1 . 

t (b)  you  are  confident  of  their  feelings  of 

dissatisfaction. 

(c)  you  consider  amenable  to  participation 
(time  required  : 20  - 25  minutes). 

2.  List  those  names  in  the  appropriate  classification  section  of  the 
attached  data  sheets. 

3.  Complete  the  additional  information  indicated  on  the  data  sheets. 

4.  (Optional)  Call  the  individuals  and  personally  ask  them  to  part- 
icipate in  the  research  project.  Indicate  to  them  that  they  will 
be  contacted  by  Sammy  Pearson  within  two  days  to  complete  a ques- 
tionnaire. 

5.  If  an  individual  declines  to  participate,  remove  the  name  from 
the  list. 

6.  List  at  least  six  names  (more  is  preferable)  in  each  of  the  three 
classifications. 

7.  Return  the  completed  data  sheets  to  either  Dr.  James  Bailey  or 
Sammy  Pearson. 


♦ Thank  you  very  much  for  your  cooperation. 


satisfaction  or 
in  this  research 


1- 


313 


Appendix  5.5 
FACTOR- DIMENS ION  MATRIX 

The  following  computer  listing  illustrates  the  complete 
factor  loading  from  a SPSS  factor  analysis  of  the  29  subject 
responses  to  the  measurement  instruments.  The  variables, 
identified  as  R01  to  R39,  correspond  to  the  user  satisfac- 
tion factors  shown  in  Appendix  3.8. 
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